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DOD  Partially  Used  Best  Practicesfor  Analyzing 
Alternatives  and  Should  Do  So  Fully  for  Future 
Military  Construction  Decisions 


Why  GAO  Did  This  Study 

DOD’s  JIAC,  currently  located  at  RAF 
Molesworth  in  the  United  Kingdom, 
provides  critical  intelligence  support  for 
the  U.S.  European  and  Africa 
Commands  and  U.S.  allies. 

GAO  was  asked  to  review  DOD’s 
decision  to  consolidate  and  relocate 
the  JIAC  to  RAF  Croughton,  United 
Kingdom.  This  report  1)  describes  key 
considerations  that  influenced  DOD’s 
decision  and  2)  evaluates  the  extent  to 
which  DOD’s  AOA  process  for  its  JIAC 
consolidation  project  aligns  with  best 
practices  for  such  analyses. 

GAO  visited  current  JIAC  facilities  to 
observe  their  condition  and  discuss 
with  officials  how  certain  factors  affect 
the  JIAC’s  ability  to  effectively  conduct 
its  mission,  and  the  base  at  which 
DOD  plans  to  consolidate  the  JIAC. 
GAO  also  discussed  this  AOA  process 
with  officials  who  participated  in,  and 
are  knowledgeable  about,  the  process; 
reviewed  documentation  used  by  DOD 
during  the  process;  and  evaluated  the 
information  using  best  practices 
identified  by  GAO  as  necessary 
characteristics  of  a  high  quality, 
reliable  AOA  process. 

What  GAO  Recommends 

GAO  recommends  that  DOD  de\elop 
guidance  that  requires  the  use  of  best 
practices — including  those  identified  by 
GAO — when  conducting  AOA 
processes  for  certain  future  military 
construction  projects.  DOD  did  not 
agree,  stating  that  the  best  practices 
do  not  apply  to  military  construction 
project  decision-making  processes. 
GAO  continues  to  belies  that  its 
recommendation  is  valid,  as  discussed 
in  this  report. 


View  GAO-16-853.  For  more  information, 
contact  Brian  J.  Lepore  at  (202)  512-4523  or 
leporeb@gao.gov. 


What  GAO  Found 

The  Department  of  Defense’s  (DOD)  decision  to  consolidate  and  relocate  its 
Joint  Intelligence  Analysis  Complex  (JIAC)  to  Royal  Air  Force  (RAF)  base 
Croughton  in  the  United  Kingdom  (U.K.)was  influenced  by  the  JIAC’s  current 
facilities’  deteriorating  condition  and  DOD  criteria  that  guided  its  analysis  of 
alternatives  (AOA).  DOD  determined  the  current  facilities  are  not  well  suited  to 
the  JIAC’s  mission,  in  poor  condition,  and  uneconomical  to  upgrade.  In  light  of 
this,  DOD's  AOA  process  was  driven  by  five  criteria.  Two  of  these — impact  on 
intelligence  operations  and  impact  on  bilateral  and  multinational  intelligence 
collaboration — were  considered  critical,  and  three  others — impact  on 
international  agreements  and  relationships,  impact  on  community  quality  of  life, 
and  the  business  case  for  consolidation — important  but  secondary. 

DOD’s  AOA  process  for  JIAC  consolidation  fully  or  substantially  met  6  best 
practices,  partially  or  minimally  met  15,  and  did  not  meet  1.  Thus,  it  partially  met 
the  four  characteristics  encompassing  these  practices  that  GAO  has  identified  as 
necessary  to  produce  a  high  quality,  reliable  AOA  process: 

•  Well  documented:  DOD’s  AOA  process  partially  met  the  best  practice  of 
tying  benefits  or  effectiveness  to  mission  need,  with  a  general 
explanation  of  how  mission  needs  will  be  met  under  DOD’s  five  AOA 
criteria.  Flowever,  DOD  minimally  met  the  best  practice  of  documenting 
the  AOA  process  in  a  single  document. 

•  Comprehensive :  DOD’s  AOA  process  fully  met  the  best  practice  of 
defining  mission  need,  but  minimally  met  the  best  practice  of  developing 
life  cycle  cost  estimates.  Specifically,  DOD  officials  characterized  the 
level  of  detail  for  its  2011  preliminary  estimates  as  “extremely  rough.” 

•  Unbiased :  DOD’s  AOA  process  substantially  met  the  best  practice  of 
ensuring  that  the  process  was  impartial,  in  part  because  an  independent 
DOD  organization  reviewed  its  conclusions.  In  contrast,  DOD  minimally 
met  the  best  practice  of  comparing  alternatives,  as  it  did  not  provide 
evidence  that  all  alternatives  were  evaluated  against  the  same  data. 

•  Credible:  DOD’s  AOA  process  fully  met  the  best  practice  of  defining 
criteria,  but  minimally  met  the  best  practice  of  describing  alternatives. 
Specifically,  DOD’s  AOA  body  of  work  does  not  provide  sufficient  detail 
on  each  alternative  to  allow  for  robust  analysis. 

According  to  DOD  officials,  the  department  did  not  follow  best  practices  when 
conducting  its  JIAC  consolidation  AOA  process.  This  is  because  DOD  does  not 
have  guidance  outlining  a  set  of  AOA  best  practices  for  military  construction 
decisions.  Without  guidance  for  using  AOA  best  practices  during  certain  military 
construction  projects — such  as  those  above  a  certain  monetary  threshold — DOD 
and  Congress  may  face  oversight  challenges,  and  DOD  may  have  trouble 
justifying  its  decisions  for  military  construction  projects  in  the  future. 

This  is  a  public  version  of  a  classified  report  that  GAO  issued  previously.  It 
excludes  classified  information  that  provided  additional  detail  on  DOD’s  decision 
to  consolidate  the  JIAC  at  RAF  Croughton. 
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September  30,  2016 
Congressional  Requesters 

According  to  the  Department  of  Defense  (DOD),  its  Joint  Intelligence 
Analysis  Complex  (JIAC)  is  critical  to  providing  intelligence  support  for  the 
U.S.  European  and  Africa  Commands  (EUCOM  and  AFRICOM)  and  U.S. 
allies.  DOD  has  stated  that  the  JIAC’s  21  current  facilities — located  at 
Royal  Air  Force  (RAF)  base  Molesworth  in  the  United  Kingdom — are 
inadequate,  inefficient,  and  were  not  designed  to  support  the  JIAC’s 
intelligence  missions,  leading  to  costly  sustainment  challenges  and 
instances  of  degraded  theater  intelligence  capabilities.  These  include, 
according  to  the  Air  Force,  numerous  facility-related  information 
technology  system  outages  that  have  severely  degraded  theater 
command  and  control  and  intelligence  capabilities  for  several  hours  at  a 
time.  To  address  these  issues,  the  Air  Force  plans  to  spend  almost  $240 
million  for  a  military  construction  project  that  will  consolidate  and  relocate 
the  current  JIAC’s  facilities  in  order  to  provide  additional  space — designed 
to  support  the  JIAC — that  fully  enables  current  intelligence  missions.1 

According  to  DOD,  in  the  fall  of  2009,  DOD  began  its  analysis  of 
alternatives  (AOA)  process  for  consolidation  of  the  JIAC,  considering  as 
potential  alternatives  locations  in  Europe  and  the  Unites  States.  By  the 
spring  of  2013,  DOD  had  concluded  that  its  preferred  alternative  was  to 
move  the  JIAC  from  its  current  location  at  RAF  Molesworth  and 
consolidate  the  JIAC’s  facilities  at  RAF  Croughton,  both  of  which  are 
located  in  the  United  Kingdom.  To  do  so,  the  Air  Force  initiated  a  three- 
phase  military  construction  project  that  the  service  plans  to  complete  by 
the  fall  of  2020. 2  In  addition,  consolidation  of  the  JIAC  at  RAF  Croughton 


UlAC  officials  stated  that,  in  addition  to  the  cost  of  these  Air  Force-funded  facilities,  DOD 
plans  to  commit  additional  funding  to  other  supporting  facilities  and  infrastructure.  The 
officials  also  explained  that  there  will  be  additional  non-facility  costs  such  as  the  base 
operations  support  needed  to  accommodate  the  increased  logistical  requirements  that  the 
JIAC  and  its  personnel  will  place  on  RAF  Croughton’s  infrastructure.  In  addition — as 
discussed  later  in  the  report — DOD  officials  explained  that  the  department  also  plans  to 
save  money  by  closing  two  installations  in  the  course  of  consolidating  the  JIAC  at  RAF 
Croughton. 

2Because  of  DOD’s  decision  to  move  the  JIAC  to  RAF  Croughton,  the  department  plans  to 
close  both  RAF  Molesworth  and  RAF  Alconbury,  a  support  installation. 
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is  one  of  26  initiatives  that  DOD  has  decided  to  carry  out  as  part  of  the 
department’s  European  Infrastructure  Consolidation  effort.  In  January 
2015  the  department  announced  this  effort  with  the  objective  to — among 
other  things — eliminate  existing,  unneeded  facilities  to  create  long-term 
savings,  realign  existing  infrastructure  in  order  to  create  excess  that  can 
be  eliminated,  and  validate  remaining  infrastructure  for  sustaining  the 
United  States’  enduring  presence  in  Europe. 

From  2013  to  2016,  Congress  has  required  DOD  to  provide  information 
on  its  JIAC  consolidation  effort,  and  since  2013,  has  taken  a  number  of 
oversight  actions  in  the  form  of  statutory  requirements  and  committee 
direction.  For  example,  House  Report  113-102  directed  DOD  to  provide 
detailed  information  on  the  department’s  plans  for  JIAC  consolidation  at 
RAF  Croughton.3  In  other,  statutory  provisions,  Congress  prohibited  DOD 
from  obligating  or  expending  military  construction  funds  in  the  United 
Kingdom  or  awarding  construction  contracts  in  the  European  Command 
area  of  responsibility  until  DOD  provides  required  information.4  In  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2016,  Congress 
authorized  funding  for  the  JIAC  consolidation  but  prohibited  the 
expenditure  of  funds  until  the  Secretary  of  Defense  certifies  to  the 
defense  committees  and  explains  why  RAF  Croughton  remains  the 
“optimal  location”  for  the  recapitalization  of  the  JIAC.5  DOD  has  provided 
the  required  information  in  response  to  committee  direction  and  statutory 
provisions.6 

In  the  JIAC’s  Phase  1 — according  to  Air  Force  budget  documentation — 
the  Air  Force  plans  to  build  EUCOM’s  Intelligence  Analytic  Center,  a  data 
processing  center,  a  warehouse,  and  various  supporting  facilities.  This 
documentation  also  explains  that  Phase  2  would  consist  of  AFRICOM’s 
Intelligence  Analytic  Center,  the  North  Atlantic  Treaty  Organization 


3See  H.R.  Rep.  No.  113-102,  at  21 8  (2013). 

4See,  for  example,  the  National  Defense  Authorization  Act  for  Fiscal  Year  2014,  Pub.  L. 
No.  1 1 3-66,  §  2809  (2013)  and  the  Military  Construction  and  Veterans  Affairs,  and  Related 
Agencies  Appropriations  Act,  2014,  Pub.  L.  No.  113-76,  Div.  J,  Title  I  (2014). 

5See  Pub.  L.  No.  114-92,  §2310  (2015). 

6ln  response  to  the  House  Report  113-102  requirement,  DOD  provided  information  in 
August  and  December  2013.  In  response  to  the  requirement  in  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2016,  DOD  provided  its  certification  in  March  2016. 
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(NATO)  Intelligence  Fusion  Center,  a  battlefield  information  system,  and 
additional  supporting  facilities.  Phase  3  includes,  according  to  this 
documentation,  a  regional  Joint  Intelligence  Training  Facility  and  a 
physical  fitness  facility.7  In  December  2014  and  November  2015,  the 
Congress  appropriated  $92.2  million  and  $94.2  million  for  construction 
Phases  1  and  2,  respectively.8  In  August  2015,  DOD  obligated  Phase  1 
funding  to  the  United  Kingdom,  which  is  serving  as  the  executive  agent 
for  JIAC  construction.  In  July  201 6,  DOD  obligated  Phase  2  funding  to  the 
United  Kingdom.  As  of  September  2016,  DOD  officials  said  they 
anticipate  awarding  a  contract  for  Phases  1  and  2  in  mid-  2017  with 
construction  to  start  in  the  fall  of  2017.  Appendix  I  provides  a  timeline  of 
key  events  for  JIAC  Consolidation  at  RAF  Croughton. 

In  separate  requests,  we  were  asked  to  evaluate  DOD’s  process  for 
analyzing  alternatives  for  potential  JIAC  locations  and  to  evaluate  DOD’s 
process  for  estimating  costs  for  the  JIAC  consolidation  military 
construction  project  at  RAF  Croughton.  In  response,  this  report  1) 
describes  key  considerations  that  influenced  DOD’s  decision  to 
consolidate  the  JIAC  at  RAF  Croughton  and  2)  evaluates  the  extent  to 
which  DOD’s  analysis  of  alternatives  process  for  its  JIAC  consolidation 
project  aligns  with  best  practices  for  such  analyses.  In  a  separate  review, 
we  are  evaluating  the  extent  to  which  DOD’s  cost  estimation  guidance 
used  for  the  cost  estimate  of  JIAC  consolidation  at  RAF  Croughton  aligns 
with  best  practices,  how  the  department  has  assessed  other  locations  in 
Europe  for  consolidation  of  the  JIAC,  and  other  related  issues.  We  plan 
on  completing  this  review  later  in  2016. 

To  conduct  our  work,  we  assessed  information  from  various  DOD 
organizations,  including  U.S.  Air  Force  Headquarters,  the  Basing 
Directorate  of  the  Office  of  the  Secretary  of  Defense,  and  the 
headquarters  of  both  EUCOM  and  AFRICOM.  In  addition,  we  visited 
various  components  of  the  JIAC  at  RAF  Molesworth,  the  JIAC’s  support 
facilities  located  at  RAF  Alconbury,  and  the  planned  future  JIAC  site  at 
RAF  Croughton. 


7Later  in  this  report,  we  provide  more  information  on  these  organizations. 

8DOD’s  estimate  for  Phase  3  is  $53.1  million,  bringing  the  total  estimated  cost  for  the  JIAC 
consolidation  military  construction  project  at  RAF  Croughton  to  about  $240  million. 
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To  describe  the  considerations  that  influenced  DOD’s  decision  to 
consolidate  the  JIAC  at  RAF  Croughton,  we  reviewed  various  DOD 
documents  describing  the  condition  of  the  JIAC’s  current  facilities, 
including  Air  Force  budget  justification  documents  submitted  to  the 
Congress  for  the  JIAC  consolidation  military  construction  projects.  During 
our  site  visit  to  the  various  components  of  the  JIAC  at  RAF  Molesworth 
and  its  support  facilities  located  at  RAF  Alconbury,  we  met  with  JIAC 
officials  to  learn  about  the  facilities’  condition,  discussing — among  other 
things — how  their  deteriorating  condition  causes  fiscal  and  operational 
impacts.  In  order  to  describe  the  five  criteria  DOD  applied  in  its  decision 
to  consolidate  the  JIAC  at  RAF  Croughton — and  key  issues  related  to 
those  criteria — we  reviewed  DOD  documentation  from  its  AOA  process, 
such  as  a  201 1  preliminary  AOA  summary  of  the  process;  interviewed 
officials  from  DOD’s  AOA  team;  and  reviewed  additional,  related 
documents,  such  as  Air  Force  guidance,  that  address  requirements  for 
the  use  of  temporary  facilities.9 

To  determine  the  extent  to  which  DOD’s  analysis  of  alternatives  process 
for  its  JIAC  consolidation  project  aligns  with  best  practices  for  such 
analyses,  we  reviewed  all  data  and  documentation  developed  by  DOD  as 
a  part  of  its  AOA  process  from  the  initial  concept  proposal  of  JIAC 
consolidation  in  the  fall  of  2009  to  the  Resource  Management  Decision 
for  JIAC  consolidation,  issued  by  the  Secretary  of  Defense  in  April  2013. 
We  refer  to  this  collection  of  information  (produced  from  the  fall  of  2009  to 
the  spring  of  2013)  as  DOD’s  “AOA  body  of  work.”  In  this  study,  we  cite 
key  documents  from  DOD’s  body  of  work  with  abbreviated  titles.  For 
example,  in  the  fall  of  201 1 ,  EUCOM  produced  a  preliminary  summary  of 
DOD’s  AOA  efforts — up  to  that  point  in  the  process — named  Analysis  of 
Alternatives  (AOA)  for  Recapitalization  of  Intelligence  Facilities  at  RAF 
Molesworth.  We  refer  to  this  document  as  the  “201 1  preliminary  AOA 
summary.”  In  addition,  we  discussed  DOD’s  AOA  process  for  JIAC 
consolidation  with  officials  from  organizations  in  the  AOA  team  and  at  the 
current  JIAC.  After  collecting  available  data  and  documentation  from 
DOD,  we  evaluated  DOD’s  AOA  body  of  work  against  GAO’s  22  AOA 


9Air  Force  Instruction  32-1032,  Planning  and  Programming  Appropriated  Fund 
Maintenance,  Repair,  and  Construction  Projects,  (Sep.  24,  2015). 
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best  practices.10  We  then  scored  DOD’s  AOA  body  of  work  against  each 
best  practice.11  The  team  used  the  average  of  the  scores  for  each  of  the 
individual  best  practices  to  determine  an  overall  score  for  the  four 
summary  characteristics  for  a  high-quality  reliable  AOA  process —  well- 
documented,  comprehensive,  unbiased,  and  credible.12  Next  we  shared 
our  draft  analysis  with  DOD,  asking  that  the  department  provide  technical 
comments  and  any  additional  documentation  that  might  impact  our 
assessment.  We  then  incorporated  these  additional  comments  to  ensure 
our  analysis  included  all  available  information.  Finally,  we  applied  the 
same  methodology  and  scoring  process  explained  above  to  revise  our 
initial  analysis  based  on  DOD’s  technical  comments  and  any  additional 
evidence  received.  For  those  characteristics  of  the  AOA  process  that 
received  a  lower-than-average  score,  we  met  with  DOD  officials  to 
discuss  potential  reasons  why  they  did  not  conform  to  best  practices  for 
those  parts  of  the  AOA  process.  In  the  course  of  applying  our  AOA  best 
practices  to  information  in  DOD’s  AOA  body  of  work,  we  assessed  the 
reasonableness  of  the  information  we  collected.  Examining  DOD’s  AOA 
process  for  JIAC  consolidation  with  our  AOA  best  practices  allowed  us  to 


1 0 

These  best  practices  were  originally  published  in  GAO,  Amphibious  Combat  Vehicle: 
Some  Acquisition  Activities  Demonstrate  Best  Practices;  Attainment  of  Amphibious 
Capability  to  Be  Determined,  GAO-16-22,  (Washington,  D.C.:  Oct.  28,  2015)  and  are 
based  on  previously  published  best  practices.  To  evaluate  DOD’s  AOA  process,  we  took 
the  following  steps:  (1)  two  GAO  analysts  separately  examined  the  AOA  information 
received  from  DOD,  providing  a  score  for  each  of  the  22  best  practices;  (2)  a  third  GAO 
analyst  adjudicated  any  differences  between  the  two  analysts’  initial  scoring;  (3)  a  GAO 
specialist  on  AOA  best  practices,  independent  of  the  audit  team,  reviewed  the  team’s 
adjudicated  AOA  documentation  and  scores,  cross-checking  the  scores  and  all  of  the 
analyses  for  consistency.  The  best  practices’  applicability  to  DOD’s  military  construction 
process  is  discussed  in  more  detail  below. 

^GAO’s  best  practices  define  five  different  qualitative  and  quantitative  categories  for 
scoring.  The  qualitative  categories  are  as  follows.  Fully  Meets:  DOD  provided  complete 
evidence  that  satisfies  the  elements  of  the  best  practice;  Substantially  Meets:  DOD 
provided  evidence  that  satisfies  a  large  portion  of  the  elements  of  the  best  practice; 
Partially  Meets:  DOD  provided  evidence  that  satisfies  about  half  of  the  elements  of  the 
best  practice;  Minimally  Meets:  DOD  provided  evidence  that  satisfies  a  small  portion  of  the 
elements  of  the  best  practice;  and  Does  Not  Meet:  DOD  provided  no  evidence  that 
satisfies  any  of  the  elements  of  the  best  practice.  The  corresponding  quantitative 
categories  are  as  follows.  Not  Met  =  1,  Minimally  Met  =  2,  Partially  Met  =  3,  Substantially 
Met  =  4,  and  Fully  Met  =  5. 

12The  resulting  average  score,  for  each  characteristic,  corresponds  to  one  of  the  five 
qualitative  categories,  as  follows:  Not  Met  =  1.0  to  1.4,  Minimally  Met  =  1.5  to  2.4,  Partially 
Met  =  2.5  to  3.4,  Substantially  Met  =  3.5  to  4.4,  and  Fully  Met  =  4.5  to  5.0. 
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assess  the  strengths  and  weaknesses  of  the  department’s  process.  Our 
best  practices  were  not  used  to  determine  whether  DOD  made  the  correct 
decision  on  the  location  for  JIAC  consolidation  or  whether  the  department 
would  have  arrived  at  a  different  conclusion  had  it  more  fully  conformed  to 
our  best  practices.  Rather,  we  used  our  best  practices  to  assess  the 
degree  to  which  DOD  can  provide  reasonable  assurance  that  its  process 
met  each  of  the  four  characteristics  of  a  high-quality,  reliable  AOA 
process.  Finally,  we  applied  the  same  methodology  and  scoring  process 
explained  above  to  revise  our  initial  analysis  based  on  DOD’s  technical 
comments  and  any  additional  evidence  received.  We  determined  that  the 
information  from  DOD’s  AOA  body  of  work  is  sufficiently  reliable  for  the 
purposes  of  describing  DOD’s  rationale  for  choosing  RAF  Croughton  as 
the  location  for  JIAC  consolidation  and  comparing  DOD’s  AOA  process  to 
our  22  best  practices  for  a  reliable  AOA  process.  More  information  on  the 
scope  and  methodology  of  our  research  is  provided  in  appendix  II,  further 
details  on  GAO’s  22  AOA  best  practices  can  be  found  in  appendix  III,  and 
we  provide  a  summary  of  our  analysis  of  DOD’s  AOA  process  for  JIAC 
consolidation — according  to  the  22  best  practices — in  appendix  IV. 

We  conducted  this  performance  audit  from  October  2015  to  September 
2016,  in  accordance  with  generally  accepted  government  auditing 
standards.  These  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 

This  report  is  a  public  version  of  a  classified  report  (GAO-1 6-563C) 
issued  in  July  2016.  DOD  deemed  some  of  the  information  in  the 
classified  report  Secret  Not  Releasable  to  Foreign  Nationals 
(SECRET//NOFORN)  and  Confidential.  Therefore,  this  report  excludes 
classified  information  that  provided  additional  detail  on  DOD’s  decision  to 
consolidate  the  JIAC  at  RAF  Croughton.  Although  the  information 
provided  in  this  public  report  is  more  limited,  it  addresses  the  same 
objectives  as  the  classified  report  and  includes  the  same 
recommendation.  Also,  the  methodology  used  for  both  reports  is  the 
same. 
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Background 


In  2006,  a  Joint  Intelligence  Operations  Center  Execute  Order  directed 
the  creation  of  joint  intelligence  operations  centers.13  According  to  DOD 
guidance,  the  role  of  these  centers  is  to  support  each  of  DOD’s 
geographical  combatant  commands  and  certain  other  DOD 
organizations. 14  The  joint  intelligence  operations  center  is  the  focal  point 
for  the  combatant  command’s  intelligence  planning,  collection 
management,  analysis,  and  production  effort.  Further,  according  to 
DOD’s  guidance,  the  center  is  to  be  organized  in  a  manner  best  suited  to 
satisfy  the  combatant  commander’s  intelligence  requirements  and 
prerogatives.  The  primary  responsibility  of  a  joint  intelligence  operations 
center  is  to  integrate  all  DOD  intelligence  from  external  defense  and 
national  intelligence  organizations,  multinational  or  partner  nations,  non¬ 
governmental  organizations,  other  government  departments  and 
agencies,  and  law  enforcement  organizations  to  ensure  that  accurate, 
timely,  and  complete  intelligence  is  available  to  positively  affect 
combatant  command  operations.  In  addition,  DOD’s  guidance  outlines 
other  responsibilities. 

•  Supporting  joint  operation  planning  and  conducting  intelligence 
operations  in  support  of  the  combatant  commander  and  staff  and 
subordinate  component  commands,  among  other  organizations. 

•  Maintaining  visibility  on  all  intelligence  collection  resources  available 
to  the  command,  aiding  the  combatant  commander  and  staff  in 
determining  intelligence  gaps  and  shortfalls  in  intelligence  collection 
capability,  and  recommending  solutions  to  mitigate  them. 

•  Ensuring  timely  support  of  the  combatant  commander  by  submitting 
requests  to  intelligence  community  production  centers. 

The  JIAC  contains  three  intelligence  centers — one  supporting  EUCOM,  a 
second  supporting  AFRICOM,  and  a  third  supporting  NATO — as  well  as 
several  other  intelligence  organizations  that,  according  to  DOD 
documents  and  discussions  with  DOD  officials,  perform  various  functions. 


13Joint  Intelligence  Operations  Center  (JIOC)  Execute  Order  (EXORD),  April  3,  2006. 

14Joint  Chiefs  of  Staff,  Joint  Publication  2-01  Joint  and  National  Intelligence  Support  to 
Military  Operations  (Jan.  5,  2012).  In  addition  to  the  six  geographical  combatant 
commands,  the  United  States  Cyber  Command — a  sub-unified  command  under  United 
States  Strategic  Command — and  the  United  States  Forces  Korea — a  sub-unified 
command  under  United  States  Pacific  Command — operate  joint  intelligence  operations 
centers. 
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•  EUCOM  Joint  Intelligence  Operations  Center  -  Europe  Analysis 
Center:  Executes  intelligence  operations  that  are  fully  synchronized 
and  integrated  with  theater  component,  national,  and  partner  nation 
organizations,  enables  EUCOM  planning  and  execution,  and 
enhances  senior  leaders’  decision-making  across  the  entire  spectrum 
of  military  operations. 

•  AFRICOM  Directorate  for  Intelligence,  RAF  Molesworth:  Manages 
and  executes  all  aspects  of  defense  intelligence  for  the  command,  to 
include  protecting  U.S.  personnel  and  facilities,  preventing  and 
mitigating  conflict,  and  building  defense  capabilities  in  order  to 
promote  regional  stability  and  prosperity. 

•  NATO  Intelligence  Fusion  Center:  Supports  NATO  by  providing 
intelligence  to  warn  of  potential  crises  and  to  support  the  planning  and 
execution  of  the  organization’s  operations,  including  direct  intelligence 
support  to  the  organization’s  special  operations  forces. 

•  Regional  Joint  Intelligence  Training  Facility:  Trains  students  from 
EUCOM,  AFRICOM,  and  North  Atlantic  Treaty  Organization  nations, 
including  the  United  Kingdom. 

•  United  States  Battlefield  Information  Collection  and  Exploitation 
Systems:  Plans,  builds,  and  operates  the  Coalition  Intelligence  and 
Information  Enterprise  to  provide  on-demand  coalition  information 
sharing  solutions  for  both  episodic  and  enduring  missions. 

According  to  additional  DOD  guidance,  the  Assistant  Secretary  of 
Defense  for  Energy,  Installations,  and  the  Environment  is  responsible  for 
overseeing  various  aspects  of  the  department’s  military  construction 
efforts.15  These  responsibilities  include — among  other  things — monitoring 
the  execution  of  the  military  construction  program  to  ensure  the  most 
efficient,  expeditious,  cost-effective  accomplishment  of  the  program,  and 
issuing  guidance  for  the  implementation  of  DOD  military  construction 
policy. 

Other  DOD  organizations,  including  U.S.  Air  Force  Headquarters,  the 
Basing  Directorate  of  the  Office  of  the  Secretary  of  Defense,  and  the 
headquarters  of  both  EUCOM  and  AFRICOM,  made  up  DOD’s  “AOA 
team”  for  the  JIAC  consolidation  AOA  process.  This  a  term  that  we  use  to 


15DOD  Directive  4270.5,  Military  Construction  (Feb.  12,  2005). 
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describe  the  key  organizations  that  contained  the  subject  matter  experts 
who  were  involved  in  the  day-to-day  work  of  the  AOA  process  and 
worked  to  develop  the  analysis  that  is  the  foundation  of  the  AOA 
process.16  Data  and  documentation  were  developed  by  DOD’s  AOA  team 
as  a  part  of  its  AOA  process,  from  the  initial  concept  proposal  of  JIAC 
consolidation  in  the  fall  of  2009  to  the  Resource  Management  Decision 
for  JIAC  consolidation,  issued  by  the  Secretary  of  Defense  in  April  2013. 
We  refer  to  this  collection  of  information  (produced  from  the  fall  of  2009  to 
the  spring  of  2013)  as  DOD’s  “AOA  body  of  work.” 


Deteriorating 
Condition  of  Current 
JIAC  Facilities  and 
Intelligence 
Collaboration  with 
European  Partners 
Influenced  DOD’s 
Decision  to 
Consolidate  the  JIAC 
at  RAF  Croughton 


According  to  DOD  documents  and  discussions  with  department  officials, 
DOD  decided  to  consolidate  the  JIAC’s  constituent  organizations  in  new 
facilities  designed  to  support  the  JIAC’s  intelligence  missions  because 
DOD  had  determined  that  the  JIAC’s  current  facilities  at  RAF  Molesworth 
are  not  designed  to  support  these  missions  and  are  in  poor  condition, 
resulting  in  both  financial  and  operational  impacts.  Through  an  AOA 
process  in  which  it  considered  a  range  of  alternatives,  DOD’s  AOA  team 
recommended  RAF  Croughton  as  the  preferred  alternative  for  JIAC 
consolidation,  prioritizing  intelligence  collaboration  with  the  United 
Kingdom  and  relationships  with  European  partner  nations  as  the  two  most 
critical  factors. 


16Over  the  course  of  our  review,  we  asked  for  original  data  and  documentation  that  the 
AOA  team  used  in  its  analyses.  In  those  instances  where  DOD  officials  stated  that  these 
data  or  documentation  were  not  available,  we  asked  to  speak  with  DOD  officials  who 
participated  in  the  AOA  process.  DOD  was  able  to  provide  several  officials  who 
participated  in  the  process.  However,  this  was  not  always  the  case.  According  to  DOD 
officials  with  whom  we  spoke,  certain  personnel  who  may  have  participated  in  the  AOA 
process  were  not  available.  The  officials  provided  various  reasons,  such  as  contractors 
who  no  longer  worked  for  the  relevant  DOD  organization,  civilians  who  had  retired,  or 
military  personnel  who  rotated  to  different  assignments.  DOD  officials  explained  that, 
given  the  AOA  process  began  in  2009  and  military  personnel  typically  rotate  on  3-year 
assignments,  these  personnel  could  be  on  their  second  or  third  rotation  since  working  as 
part  of  the  AOA  team. 


Page  9 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


DOD  Has  Determined 
That  the  JIAC’s  Current 
Facilities  Are 
Deteriorating,  Producing 
Financial  and  Operational 
Impacts 

RAF  Molesworth  Facilities  Not 
Designed  to  Support  Its 
Missions 


Financial  Impacts  of 
Deteriorating  and  Temporary 
Facilities 


Operational  Impacts  of 
Deteriorating  Facilities 


According  to  DOD  documents  and  discussions  with  JIAC  officials  at  the 
JIAC’s  current  facilities  at  RAF  Molesworth,  the  current  facilities  are  (1) 
are  not  well  suited  to  the  JIAC’s  mission,  (2)  are  in  poor  condition,  and  (3) 
are  continuing  to  deteriorate.  The  facilities’  condition  has  had  both 
financial  impacts  for  DOD  and  operational  impacts  on  the  JIAC’s  ability  to 
conduct  its  intelligence  missions,  according  to  DOD  documents  and 
officials.  As  a  result,  the  department  decided  to  recapitalize  and 
consolidate  the  JIAC’s  facilities. 

In  its  budget  justification  documents  for  the  JIAC  consolidation  military 
construction  projects,  the  Air  Force  states  that  the  intelligence  mission 
being  conducted  by  the  JIAC  has — over  the  past  25  years — grown 
substantially.  According  to  the  Air  Force,  this  growth  has  created  a  severe 
shortfall  of  intelligence  spaces,  resulting  in  intelligence  organizations  that 
are  housed  in  over  21  undersized  facilities,  including  an  aircraft  hangar 
that  dates  from  World  War  II,  several  cruise  missile  facilities  that  date 
from  the  Cold  War,  and  leased,  relocatable  facilities.  The  Air  Force 
explains  that  none  of  these  facilities  were  designed  for  their  current  use, 
and  that  the  shortage  degrades  the  reliability  of  theater  and  national 
communications  and  intelligence  assets. 

In  budget  justification  documents,  the  Air  Force  stated  that  over  $90 
million  in  Intelligence  Community  funds  have  been  spent  since  2005  to 
sustain  the  JIAC’s  facilities  and  supporting  utilities  systems  in  a  condition 
that  only  minimally  supports  the  mission.  Further,  this  documentation 
explains  that  aging  and  inefficient  power  and  cooling  systems — critical  to 
the  intelligence  mission — are  not  able  to  be  economically  upgraded,  due 
to  the  nature  of  existing  facilities.  For  example,  a  report  from  the  Defense 
Intelligence  Agency — assessing  the  agency’s  facilities  at  EUCOM’s  Joint 
Intelligence  Operations  Center  at  RAF  Molesworth — recommends  that  the 
original,  25-year  old  electrical  and  mechanical  infrastructure  for  all 
buildings  should  be  replaced  because  this  infrastructure  is  older  than  its 
intended  design  life.  In  addition,  officials  from  EUCOM  and  Air  Force 
headquarters  explained  that — due  to  the  difference  in  how  the  Air  Force 
funds  the  maintenance  of  permanent  and  temporary  facilities — the 
organization  using  temporary  facilities  is  responsible  for  funding  their 
maintenance  costs.  As  a  result,  according  to  the  officials,  the  use  of  these 
temporary  facilities  strains  the  budgets  of  certain  JIAC  organizations.  We 
discuss  this  issue  further  later  in  this  report. 

According  to  JIAC  and  EUCOM  officials,  as  well  as  Air  Force 
documentation,  unanticipated  power  and  cooling  system  failures  cause 
frequent  down  time  for  intelligence  analysts,  wasting  thousands  of 
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personnel  hours  in  analytical  effort  and  exposing  EUCOM  and  AFRICOM 
to  intelligence  blackouts.  During  our  site  visit  to  the  current  JIAC  facilities 
at  RAF  Molesworth,  and  discussions  with  JIAC  officials,  we  learned  of 
several  instances  of  these  challenges.  For  example,  according  to  JIAC 
officials,  it  is  common  for  aging  fire  alarm  systems  in  the  installation’s 
buildings  to  trigger  false  alarms.  This  causes  evacuations  during  which 
personnel  cannot  conduct  intelligence  operations. 

According  to  EUCOM  officials,  in  March  2016,  a  leak  in  the  roof  of  the 
JIAC’s  Building  100  allowed  water  into  a  power  distribution  panel. 

Officials  said  that  this  leak  caused  a  fire  that  knocked  out  power  to 
several  of  the  building’s  utility  systems,  including  heating,  air  conditioning, 
ventilation  equipment,  and  sewage  pumps.  Furthermore,  the  EUCOM 
officials  explained  that  Building  100  houses  several  key  JIAC 
organizations,  including  EUCOM’s  Joint  Intelligence  Operations  Center  at 
RAF  Molesworth,  the  center’s  National  Geospatial  Intelligence  Agency 
component,  and  the  center’s  watch  floor.17  According  to  EUCOM  officials, 
the  loss  of  utility  service  disrupted  the  work  environment  of  the  building’s 
200  personnel  for  an  estimated  30  hours.  EUCOM  officials  provided  the 
following  pictures  that  detail  the  leak  and  fire  damage. 


17According  to  JIAC  officials,  on  the  watch  floor,  the  Center’s  personnel  track  key 
intelligence  topics  or  regions  in  order  to  provide  updates  or  warnings  to  the  headquarters 
supported  by  the  JIAC. 


Page  11 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


Figure  1:  Water  Leaking  onto  a  Power  Distribution  Panel  (left)  and  Scorch  Marks  Left  by  Fire  on  the  Panel  (right)  in  EUCOM 
Joint  Intelligence  Operations  Center  at  Royal  Air  Force  (RAF)  Base  Molesworth  Building  100 


Source:  U.S.  European  Command  Joint  Intelligence  Operations  Center  at  RAF  Molesworth.  |  GAO-16-853 


In  another  example,  the  aging  infrastructure  of  a  facility  originally  built 
during  the  1980s  to  store  cruise  missiles — and  now  used  by  the  JIAC  as 
office  space — has  resulted  in  operational  impacts,  according  to  JIAC 
officials.  They  explained  that  in  some  instances,  due  to  its  age,  the 
telephone  infrastructure  housed  in  this  facility  has  failed,  bringing  the 
phone  service  down  for  the  entire  installation.  Because  of  this,  according 
to  officials,  analysts  cannot  communicate  with  other  analysts  outside  of 
their  office  space. 

In  addition,  JIAC  officials  provided  information  regarding  the  impact  of 
current  buildings’  condition  on  facility  security.  They  explained  that  there 
are  12  sensitive  compartmented  information  facilities  operating  on  RAF 
Molesworth.18  The  officials  stated  that  while  all  12  comply  with  intelligence 


1 8 

According  to  Director,  National  Intelligence,  Intelligence  Community  Directive,  Number 
705,  Sensitive  Compartmented  Information  Facilities  (May  26,  2010),  all  information 
classified  as  sensitive  compartmented  information  must  be  processed,  used,  stored  or 
discussed  in  an  accredited  sensitive  compartmented  information  facility. 
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community  guidance,  recurring  maintenance  at  RAF  Molesworth  impacts 
operations  by  requiring  escorts,  suspends  operations  while  maintenance 
is  conducted  by  foreign  nationals,  and  poses  a  risk  for  increased 
surveillance  by  adversaries.  According  to  JIAC  officials,  JIAC  facilities’ 
age  can  lead  to  unexpected,  security-related  operational  impacts.  For 
instance,  according  to  JIAC  officials,  at  the  1980s  cruise  missile  storage 
facility,  several  rounds  of  upgrades  mean  that  the  building  is  packed  with 
multiple  layers  of  electrical  wiring,  making  current  renovations 
complicated.  JIAC  officials  told  us  that  recently  contractors  removed  a 
panel  that — unbeknownst  to  them — controlled  the  building’s  security 
systems.  This  led  to  the  systems’  failure,  an  important  operational  impact 
for  a  facility  that  houses  intelligence  organizations. 


DOD  Considered 
Alternatives  and 
Determined  That  RAF 
Croughton  Best  Meets 
Criteria  DOD  Identified  for 
JIAC  Consolidation 


According  to  DOD  documents  and  discussions  with  DOD  officials,  DOD’s 
AOA  team  conducted  an  AOA  process  in  which  the  team  considered  a 
range  of  facilities  and  alternatives  and  recommended  RAF  Croughton  as 
the  preferred  alternative  for  JIAC  consolidation.19  In  doing  so,  the  AOA 
team  prioritized  intelligence  operations,  along  with  bilateral  and 
multinational  intelligence  collaboration  as  the  two  most  critical  factors 
influencing  their  decision. 


DOD  Consideration  of  Existing 
Permanent  or  Temporary 
Facilities  to  Use  for  JIAC 
Consolidation 


One  of  the  options  included  in  the  AOA  process  was  modernization  of 
existing  facilities  where  feasible  and  construction  of  new  facilities  at  RAF 
Molesworth.  However,  officials  from  the  Office  of  the  Secretary  of 
Defense,  Air  Force  Headquarters,  and  EUCOM  explained  that  the  AOA 
team  did  not  feel  that  renovation  of  the  JIAC’s  existing  permanent  and 
temporary  facilities  was  feasible,  given  statutory  requirements  and  service 
guidance.  Specifically,  the  officials  stated  that  according  to  U.S.  law 
governing  the  use  of  operations  and  maintenance  funding  to  repair  a 
DOD  facility,  if  the  current  estimate  of  the  cost  of  the  repair  project 
exceeds  75  percent  of  the  estimated  cost  of  a  military  construction  project 


1 Q 

As  discussed  previously  in  this  report,  we  assessed  information  from  various  DOD 
organizations,  including  U.S.  Air  Force  Headquarters,  the  Basing  Directorate  of  the  Office 
of  the  Secretary  of  Defense,  and  the  headquarters  of  both  EUCOM  and  AFRICOM.  These 
organizations  made  up  DOD’s  “AOA  team”  for  the  JIAC  consolidation  AOA  process.  This 
a  term  that  we  use  to  describe  the  key  organizations  that  contained  the  subject  matter 
experts  who  were  involved  in  the  day-to-day  work  of  the  AOA  process  and  worked  to 
develop  the  analysis  that  is  the  foundation  of  the  AOA  process. 
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to  replace  the  facility,  the  department  must  provide  reasons  why 
replacement  of  the  facility  is  not  in  the  best  interest  of  the  government.20 

The  officials  explained  that,  given  the  challenges  of  renovating  the  JIAC’s 
existing  permanent  facilities,  the  AOA  team  believed  replacement  of  the 
JIAC’s  current  facilities  was  in  the  best  interest  of  the  government.  In 
addition,  two  temporary  facilities  are  used  to  house  AFRICOM’s 
Directorate  for  Intelligence  organization  and  the  NATO  Intelligence  Fusion 
Center.  Continued  use  of  these  facilities  is  not  viable,  according  to 
officials,  due  to  both  Air  Force  guidance  and  budgetary  considerations. 
According  to  this  Air  Force  guidance,  temporary  facilities  should  be  used 
for  the  shortest  term  possible  (fewer  than  5  years)  and  are  used  to  fulfill  a 
temporary  requirement.21  Officials  stated  that  these  facilities  have  housed 
the  Directorate  for  Intelligence  and  NATO  Intelligence  Fusion  Center  for 
longer  than  5  years.  Further,  additional  guidance  holds  that — for 
temporary  facilities — the  organization  using  them  is  responsible  for 
funding  their  maintenance  costs  out  of  the  organization’s  operations  and 
maintenance  budget.22  According  to  officials,  this  is  in  contrast  to 
permanent  facilities’  sustainment  funding,  which  is  provided  by  Air  Force 
headquarters.  As  a  result,  according  to  the  officials,  JIAC  organizations 
must  pay  for  the  sustainment  of  these  temporary  facilities  using  their 
operations  and  maintenance  budget,  although  this  budget  is  not 
resourced  to  cover  these  costs.  As  a  result,  the  officials  explained  that  the 
use  of  these  temporary  facilities  strains  the  JIAC  organizations’ 
operations  and  maintenance  budget. 

According  to  officials  from  the  Office  of  the  Secretary  of  Defense,  Air 
Force  Headquarters,  and  EUCOM,  DOD’s  decision  to  consolidate  the 
JIAC’s  organizations  in  a  new  facility  designed  for  its  mission  was  also 
predicated  on  the  lack  of  existing,  permanent  sensitive  compartmented 
office  space  available  in  other  locations.  The  JIAC’s  organizations  require 
a  total  of  239,315  square  feet  of  secure  office  space  to  conduct  their 
intelligence  missions.  The  officials  stated  that  DOD  had  not  conducted 


20See  10  USC  §  2811  (d)(2). 

21  Air  Force  Instruction  32-1032,  Planning  and  Programming  Appropriated  Fund 
Maintenance,  Repair,  And  Construction  Projects  (Sept.  24,  2015). 

22Air  Force  Instruction  32-1021 ,  Planning  and  Programming  Military  Construction 
(MILCON)  Projects  (Feb.  25,  2016). 
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specific  analysis  in  support  of  this  conclusion,  but  that  they  believed  no 
other  existing  DOD  installation  had  a  facility  with  enough  empty  sensitive 
compartmented  space  into  which  the  JIAC’s  organizations  could  move. 

According  to  officials  from  the  Office  of  the  Secretary  of  Defense,  Air 
Force  Headquarters,  and  EUCOM — based  on  the  assumption  that  DOD 
would  construct  a  new  facility  designed  to  support  the  JIAC’s  intelligence 
missions — in  2009  the  department  began  its  AOA  process  to  select  a 
location  for  JIAC  consolidation.  According  to  officials  from  the  Office  of 
the  Secretary  of  Defense  and  EUCOM,  at  the  beginning  of  the  process, 
members  of  DOD’s  AOA  team  developed  five  AOA  criteria  that  were 
based  on  their  subject  matter  expertise  concerning  the  EUCOM  and 
AFRICOM  theaters  and  the  JIAC’s  mission.  DOD’s  AOA  team  assessed 
alternatives  using  these  five  criteria,  two  of  which  it  deemed  critical.23  The 
critical  criteria  were  (1)  impact  on  intelligence  operations  and  (2)  impact 
on  bilateral  and  multinational  intelligence  collaboration;  the  other  criteria 
were  (3)  impact  on  international  agreements  and  relationships;  (4)  impact 
on  community  quality  of  life;  and  (5)  business  case  (based  on  estimated 
military  construction  costs).  The  following  is  a  summary — based  on  our 
review  of  DOD  documents  and  discussions  with  department  officials — of 
DOD’s  rationale  for  the  choice  of  RAF  Croughton  as  the  alternative  that 
best  meets  these  five  criteria. 


DOD’s  First  Critical  Criterion 
Was  How  JIAC  Consolidation 
Would  Impact  Intelligence 
Operations 


According  to  a  201 1  summary  of  preliminary  AOA  findings,  the  location  of 
the  JIAC  has  little  impact  on  many  aspects  of  its  intelligence  operations. 
For  example,  the  document  states  that  current  information  technology 
and  networks  enable  the  intelligence  community  to  rapidly  analyze, 
synthesize,  and  share  intelligence  over  great  distances.  However,  DOD’s 
AOA  team  concluded  that  the  JIAC’s  location  in  Europe  has  significant 
impact  on  at  least  three  key  mission  areas:  synchronization  with  the 
headquarters  supported  by  the  JIAC,  delivery  of  regional  intelligence 
training,  and  access  to  non-U. S.  regional  experts. 


First,  according  to  DOD  officials  and  documents,  the  AOA  team  believed 
that  keeping  the  JIAC  in  a  time  zone  as  close  as  possible  to  the  time  zone 
of  the  headquarters  organizations  that  it  supports  would  facilitate  the  most 


23Below,  we  discuss  how  DOD’s  AOA  team  evaluated  these  alternatives  during  the 
department’s  AOA  process  for  JIAC  consolidation. 
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effective  and  efficient  operations.  Officials  from  Air  Force  Headquarters, 
EUCOM,  the  Office  of  the  Secretary  of  Defense,  and  the  JIAC  stated  that 
DOD  had  not  conducted  specific  analysis  in  support  of  this  conclusion. 
However,  the  201 1  preliminary  AOA  summary  states  that  the  JIAC’s 
current  location  in  the  United  Kingdom  means  that  the  work  schedules  of 
JIAC  personnel  are  closely  aligned  with  EUCOM  and  AFRICOM 
headquarters.  In  addition,  according  to  an  example  in  documentation 
provided  by  EUCOM,  locating  the  JIAC  in  the  United  Kingdom’s  time 
zone  provides  operational  benefits.  Specifically,  it  allows  the  JIAC’s 
personnel  to  consistently  interact  with  these  headquarters  during  the 
same  work  day,  thus  providing  “complete  integration”  of  the  JIAC’s 
intelligence  support  into  the  decision  making  process  of  EUCOM’s 
commander.  According  to  the  201 1  preliminary  AOA  summary,  a 
Stuttgart,  Germany  location  for  the  JIAC  organizations  supporting  the 
EUCOM  and  AFRICOM  headquarters  would  essentially  eliminate 
distance-related  synchronization  and  coordination  challenges  between 
the  JIAC  and  these  headquarters,  both  of  which  are  located  in  Stuttgart. 
Further,  locating  the  NATO  Intelligence  Fusion  Center  in  Benelux  would 
do  the  same  for  the  headquarters  it  supports.  24  However,  DOD  officials 
provided  two  reasons  why  DOD’s  AOA  team  determined  that  a  location  in 
the  United  Kingdom  was  a  better  alternative  than  a  location  in  either 
Stuttgart  or  Benelux.  According  to  officials,  current  facilities  in  Stuttgart 
are  already  filled  to  above  capacity  and  thus  cannot  accommodate  the 
JIAC’s  organizations.  Also,  when  compared  to  locations  in  the  United 
Kingdom,  these  other  European  locations  would  not  meet  some  of  DOD’s 
other  AOA  criteria  as  well,  as  discussed  below. 

The  second  mission  area  considered  by  the  AOA  team  was  intelligence 
training.  JIAC  officials  explained  that  the  United  Kingdom  locations 
included  in  DOD’s  AOA  process  provide  the  advantage  of  being  close  to 
multiple  international  airports  located  in  the  London  area.  According  to  an 
official  from  the  JIAC’s  Regional  Joint  Intelligence  Training  Facility,  this  is 
important  given  that  in  fiscal  year  2015,  the  facility’s  instructors  traveled 
outside  of  the  United  Kingdom  to  multiple  countries  in  order  to  teach  80 
courses,  or  44  percent  of  the  facility’s  total  course  load. 


24“Benelux”  is  a  term  that  refers  to  the  area  consisting  of  Belgium,  the  Netherlands,  and 
Luxembourg. 
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Access  to  non-U. S.  regional  experts  was,  according  to  DOD  officials,  the 
third  mission  area  on  which  the  AOA  team  focused.  JIAC  officials 
explained  that  a  United  Kingdom  location  for  the  JIAC  provides 
unparalleled  access  to  non-U. S.  experts.  According  to  U.S.  intelligence 
community  guidance,  intelligence  organizations  are  required  to  engage 
with  entities  outside  of  the  community  in  order  to — among  other  things — 
explore  alternative  perspectives  and  gain  insights.25  According  to  the 
201 1  preliminary  AOA  summary,  locations  in  Europe  are  well  suited  to 
providing  JIAC  organizations  with  frequent  and  sustained  personal 
interaction  with  foreign  experts  in  Europe.  Officials  from  EUCOM,  the 
Office  of  the  Secretary  of  Defense,  and  the  Air  Force  explained  that 
DOD’s  AOA  team  did  not  conduct  specific  analysis  in  support  of  this 
conclusion.  However,  JIAC  officials  explained  that  the  United  Kingdom 
has  unique  value  as  a  JIAC  location  because  most  experts  in  the  United 
Kingdom  speak  English  and  so  are  more  accessible  to  JIAC  personnel. 


DOD’s  Second  Critical 
Criterion  Was  How  JIAC 
Consolidation  Would  Impact 
Bilateral  and  Multinational 
Intelligence  Collaboration 


According  to  our  review  of  DOD  documentation  and  discussions  with 
department  officials,  the  DOD  AOA  team  believed  that  United  Kingdom 
locations  facilitate  the  highest-quality  intelligence  collaboration  between 
U.S.  and  allied  intelligence  personnel.  The  201 1  preliminary  AOA 
summary  states  that  while  technology  enables  digital  conferencing, 
technology  cannot  fully  enable  some  of  the  more  nuanced  aspects  of 
developing  close  professional  relationships  and  trust.  According  to 
officials  from  EUCOM,  the  Office  of  the  Secretary  of  Defense,  and  the  Air 
Force,  DOD’s  AOA  team  did  not  conduct  specific  analysis  in  support  of 
this  conclusion.  However,  JIAC  officials  stated  they  strongly  believe  that, 
for  intelligence  work,  trust  is  the  single  most  important  factor  in  achieving 
high  value  collaboration,  and  that  in-person  communication  between 
intelligence  analysts  from  the  United  States  and  partner  nations  facilitates 
the  development  of  closer  personal  relationships,  which,  in  turn,  builds 
trust.  In  regard  to  bilateral  intelligence  collaboration,  the  officials 
explained  that  JIAC  locations  in  the  United  Kingdom  are  particularly  well 
suited  to  meeting  this  criterion  because  the  United  Kingdom  is  the  United 
States’  most  capable  and  important  intelligence  ally.  For  example,  key 
United  Kingdom  intelligence  facilities  are  located  at  RAF  Wyton,  about  66 
miles  from  RAF  Croughton,  which  would  enable  frequent  in-person 


25Director,  National  Intelligence,  Intelligence  Community  Directive  205,  Analytic  Outreach 
(Aug.  28,  2013). 
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DOD’s  Third  Criterion  Was 
How  JIAC  Consolidation  Would 
Impact  International 
Agreements  and  Relationships 


collaboration.  JIAC  officials  stated  that  United  Kingdom  personnel  located 
at  RAF  Wyton  work  on  counter-terrorism  issues,  which  are  central  to  the 
mission  of  the  portion  of  AFRICOM’s  Directorate  of  Intelligence  assigned 
to  the  JIAC,  and  analysts  from  AFRICOM  frequently  collaborate  in  person 
with  their  United  Kingdom  counterparts  at  RAF  Wyton. 

According  to  officials  from  the  Office  of  the  Secretary  of  Defense,  Air 
Force  Headquarters,  and  EUCOM,  DOD  did  not  consider  this  third 
criterion  to  be  critical,  but  did  consider  it  to  be  “quite  important.”  Key 
issues  considered  by  the  AOA  team  include  America’s  national  security 
relationship  with  the  United  Kingdom  and  the  JIAC’s  collaboration  and 
logistical  support  of  the  NATO  Intelligence  Fusion  Center.  DOD  provided 
us  with  an  August  2015  letter,  sent  to  the  chairs  and  ranking  members  of 
the  House  and  Senate  Armed  Services  committees  by  the  British 
Ambassador  to  United  States,  which  urges  the  members’  support  for 
keeping  the  JIAC  in  the  United  Kingdom  and  characterizes  “the  close 
cooperation”  between  the  two  countries  on  national  security  as  “long 
standing,  broad,  and  uniquely  deep.”  In  addition  to  the  American  -  British 
relationship,  another  issue  on  which  DOD  focused  for  this  third  criterion 
was  the  location  of  the  NATO  Intelligence  Fusion  Center. 

According  to  officials  from  the  Office  of  the  Secretary  of  Defense,  Air 
Force  Headquarters,  and  EUCOM,  one  of  the  assumptions  of  the  AOA 
process  was  that — under  any  consolidation  scenario — the  NATO 
Intelligence  Fusion  Center  would  remain  collocated  with  the  JIAC’s  other 
intelligence  organizations.  This  was  for  three  reasons. 

•  The  first  was  intelligence  collaboration.  According  to  the  201 1 
preliminary  AOA  summary,  the  center’s  colocation  with  the  JIAC’s 
EUCOM  and  AFRICOM  organizations  has  enabled  analytical 
collaboration,  enhancing  the  three  organizations’  effectiveness. 

•  The  second  reason  for  DOD’s  focus  on  the  NATO  Intelligence  Fusion 
Center’s  location  was  existing  agreements  with  the  center’s  26 
participating  nations.  According  to  the  201 1  preliminary  AOA 
summary,  the  memorandum  of  understanding  establishing  the  center 
designates  the  United  Kingdom  as  host  nation  and  requires 
unanimous  approval  of  changes  to  the  memorandum.  JIAC  relocation 
scenarios  in  which  the  NATO  Intelligence  Fusion  Center  moves 
outside  the  United  Kingdom  would  require  the  agreement  of  all  26 
participating  nations.  Further — according  to  officials  from  the  Office  of 
the  Secretary  of  Defense,  Air  Force  Headquarters,  and  EUCOM — the 
commanders  of  both  EUCOM  and  AFRICOM  were  “especially 
concerned”  that  a  U.S.  decision  to  relocate  the  JIAC  and  NATO 
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DOD’s  Fourth  Criterion  Was 
How  Consolidation  Would 
Impact  JIAC  Personnel’s 
Quality  of  Life 


Intelligence  Fusion  Center  from  the  United  Kingdom  may  provide  a 
counterproductive  signal  to  partner  nations  participating  in  the  center, 
and  could  have  resulted  in  nations  deciding  to  end  their  voluntary 
participation. 

•  The  third  reason  for  DOD’s  focus  on  the  center’s  location  was 

logistical  and  administrative  support.  According  to  JIAC  officials,  the 
memorandum  of  understanding  establishing  the  center  designates  the 
United  States  as  the  nation  that  will  provide  the  majority  of  support  to 
the  center.  Thus,  according  to  the  officials,  EUCOM’s  Joint 
Intelligence  Operations  Center  at  RAF  Molesworth  conducts  all  of  the 
logistical  and  administrative  support  services  for  both  the  AFRICOM 
Directorate  for  Intelligence  and  the  NATO  Intelligence  Fusion  Center. 
This  is  because,  when  these  two  organizations  were  created,  they 
were  intended  to  be  collocated  with  the  EUCOM  Joint  Intelligence 
Operations  Center  at  RAF  Molesworth  and  thus  did  not  require  their 
own  support  personnel  and  services.  The  officials  explained  that 
separating  the  NATO  Intelligence  Fusion  Center  from  EUCOM’s  Joint 
Intelligence  Operations  Center  at  RAF  Molesworth  would  result  in 
increased  costs  because  the  NATO  Intelligence  Fusion  Center  would 
need  to  hire  additional  personnel  to  perform  the  functions  currently 
performed  by  EUCOM  personnel.26 

According  to  the  201 1  preliminary  AOA  summary,  there  are  “long 
standing  quality  of  life  issues”  for  personnel  working  at  the  JIAC.  These 
issues  stem  in  part  from  the  deteriorating  condition  of  the  current  JIAC 
facilities  at  RAF  Molesworth,  discussed  earlier  in  this  report.  For  example, 
personnel  working  in  those  facilities  experience  frequent  summertime 
failure  of  air  conditioning  equipment.  According  to  DOD,  other  quality  of 
life  issues  stem  from  the  fact  that  the  current  JIAC  facilities  on  RAF 
Molesworth  are  geographically  separated  from  the  facilities  that  support 
JIAC  personnel,  such  as  housing  and  commissaries,  located  on  RAF 
Alconbury.  The  201 1  preliminary  AOA  summary  states  that  the  25-40 


26According  to  estimates  from  an  AFRICOM  official,  separating  the  AFRICOM  Directorate 
for  Intelligence  from  the  logistical  support  provided  by  EUCOM’s  Joint  Intelligence 
Operations  Center  would  require  AFRICOM  to  hire  approximately  24  additional 
administrative,  logistical,  and  security  personnel  in  addition  to  an  unknown  number  of 
network  support  personnel.  Further,  according  to  the  AFRICOM  official,  depending  on 
whether  the  positions  were  filled  with  only  government  civilians  and  enlisted  personnel  or 
filled  with  only  contractors,  the  cost  of  these  personnel  is  estimated  to  be  between  $2 
million  and  $5  million  per  year. 
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DOD’s  Fifth  Criterion  Was  the 
Business  Case  for  JIAC 
Consolidation 


minute  drive  between  the  two  installations  requires  personnel  to  make 
frequent  use  of  local  roads  on  which  it  can  be  challenging  for  non-United 
Kingdom  personnel  to  drive.  Thus,  the  AOA  team  assessed  AOA 
alternatives  based  on  the  degree  to  which  they  allowed  for  the  JIAC’s 
facilities  and  these  support  facilities  to  be  collocated. 

DOD  officials  stressed  that  the  AOA  team  considered  non-financial 
factors — such  as  intelligence  collaboration  and  relationships  with  partner 
nations — to  be  more  important  than  the  cost  of  JIAC  consolidation 
alternatives.  Officials  from  the  Office  of  the  Secretary  of  Defense,  Air 
Force  Headquarters,  and  EUCOM  characterized  the  AOA  process  as 
“cost  informed  but  not  cost  driven.”  However,  while  the  AOA  team  did  not 
consider  the  alternatives’  relative  cost  to  be  the  most  important  factor, 
documentation  from  the  AOA  body  of  work  and  our  discussions  with  DOD 
officials  indicate  that  the  AOA  team  did  consider  alternatives’  relative 
costs.  Thus,  the  relative  importance  of  cost  in  the  AOA  team’s  analysis  is 
somewhat  unclear.  Later  in  the  report,  we  discuss  this  issue  in  more 
detail,  examining  the  extent  to  which  the  AOA  team  followed  best 
practices  related  to  cost  estimation.27 

There  are  several  examples  of  how  the  AOA  team  addressed  the 
Business  case  criterion.  For  instance,  in  a  201 1  briefing  on  JIAC 
consolidation  alternatives,  EUCOM’s  Air  Force  component  considered 
facility  and  information  technology  costs  for  a  limited  number  of 
alternatives.  Also,  in  the  AOA  team’s  201 1  preliminary  AOA  summary,  it 
states  that  the  AOA  process  examined  estimates  of  military  construction 
costs  and  potential  savings  associated  with  recapitalization  and 
consolidation  alternatives  in  order  to  ensure  that  the  preferred  alternative 
would  achieve  the  optimum  balance  between  capital  investment  and  long¬ 
term  base  sustainment  savings.  In  addition,  throughout  our  discussions 
with  officials  from  the  Office  of  the  Secretary  of  Defense,  Air  Force 
Headquarters,  EUCOM,  and  the  JIAC,  officials  emphasized  that  the  cost 
savings  DOD  estimates  it  will  achieve  by  consolidating  the  JIAC  at  RAF 
Croughton  were  considered  throughout  the  AOA  process.  For  example, 


27ln  this  report,  we  focus  on  the  extent  to  which  DOD  met  best  practices  for  life  cycle  cost 
estimation  during  its  AOA  process.  In  a  separate  review,  we  will  examine  DOD's  post- 
AOA  cost  estimate  for  JIAC  consolidation  at  RAF  Croughton,  evaluating  the  extent  to 
which  the  department’s  estimate  aligns  with  best  practices.  We  plan  on  completing  that 
second  review  in  2016. 
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according  to  officials  from  the  Office  of  the  Secretary  of  Defense  and  the 
JIAC,  DOD’s  opportunity  to  save  money  by  closing  RAF  Molesworth,  RAF 
Alconbury,  or  both,  as  part  of  a  JIAC  consolidation  scenario  was 
considered  early  in  the  AOA  process. 


In  Its  Analysis  of 
Alternatives  for  JIAC 
Consolidation,  DOD 
Fully  or  Substantially 
Met  6  Best  Practices 
for  Conducting  Such 
an  Analysis,  but 
Partially,  Minimally,  or 
Did  Not  Meet  16  Best 
Practices 


Based  on  discussions  with  DOD  officials  and  our  review  of  documentation 
in  DOD’s  AOA  body  of  work,  we  determined  that  DOD  followed  some  but 
not  all  best  practices  during  the  JIAC  consolidation  AOA  process.28  We 
have  developed  best  practices  that  apply  to  all  AOA  processes  by  (1) 
compiling  and  reviewing  commonly  mentioned  AOA  policies  and 
guidance  used  by  different  government  and  private-sector  entities  and  (2) 
incorporating  experts’  comments  on  a  draft  set  of  practices  to  develop  a 
final  set  of  practices.  Comparing  DOD’s  AOA  process  to  our  best 
practices,  we  determined  that  DOD’s  process  partially  met  the  four 
characteristics  of  a  high-quality,  reliable  AOA  process.  DOD  officials 
stated  that  they  believe  our  AOA  practices  do  not  apply — in  their 
entirety — to  DOD’s  process  for  making  military  construction  decisions  and 
did  not  use  AOA  best  practices  to  guide  the  JIAC  consolidation  AOA 
process.  However,  we  continue  to  believe  that  our  AOA  best  practices 
should  be  used  by  DOD  when  making  military  construction  decisions. 


GAO  Developed  AOA  Best 
Practices  That  Apply  to  All 
AOA  Processes 


Because  there  was  no  single  set  of  practices  for  the  AOA  process  that 
had  been  broadly  recognized  by  both  the  government  and  private-sector 
entities,  in  2016  we  identified  22  best  practices  for  conducting  an  AOA 
process.  We  did  this  by  (1)  compiling  and  reviewing  commonly  mentioned 
AOA  policies  and  guidance  used  by  different  government  and  private- 
sector  entities  and  (2)  incorporating  and  vetting  experts’  comments  on  a 
draft  set  of  practices  to  develop  a  final  set  of  practices.  The  22  best 
practices  that  we  identified  are  grouped  into  five  AOA  phases,  as  shown 
in  table  1 . 


oo 

As  discussed  previously  in  this  report,  to  determine  the  extent  to  which  DOD’s  AOA 
process  for  JIAC  consolidation  aligns  with  best  practices  for  such  analyses,  we  reviewed 
all  data  and  documentation  developed  by  DOD  as  a  part  of  its  AOA  process  from  the  initial 
concept  proposal  of  JIAC  consolidation  in  the  fall  of  2009  to  the  Resource  Management 
Decision  for  JIAC  consolidation,  issued  by  the  Secretary  of  Defense  in  April  2013.  We 
refer  to  this  collection  of  information  (produced  from  the  fall  of  2009  to  the  spring  of  2013) 
as  DOD’s  “AOA  body  of  work.” 
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Table  1 :  GAO’s  Analysis  of  Alternatives  (AOA)  Best  Practices  Grouped  into  the  Five  Phases  of  an  AOA  Process 


AOA  phase 

Description  of  best  practices  in  phase 

Initialize  the  AOA  process 

Best  practices  that  are  applied  before  starting  the  process  of  identifying,  analyzing,  and  selecting 
alternatives.  This  includes  determining  the  mission  need  and  functional  requirements, 
developing  the  study  time  frame,  creating  a  study  plan,  and  determining  who  conducts  the 
analysis. 

Identify  alternatives 

Best  practices  that  help  ensure  the  alternatives  to  be  analyzed  are  sufficient,  diverse,  and  viable. 

Analyze  alternatives 

Best  practices  that  compare  the  alternatives  to  be  analyzed.  The  best  practices  in  this  category 
help  ensure  that  the  team  conducting  the  analysis  uses  a  standard,  quantitative  process  to 
assess  the  alternatives. 

Document  and  review  the  AOA 
process 

Best  practices  that  would  be  applied  throughout  the  AOA  process,  such  as  documenting  all 
steps  taken  to  initialize,  identify,  and  analyze  alternatives  and  to  select  a  preferred  alternative  in 
a  single  document. 

Select  a  preferred  alternative 

Best  practice  that  is  applied  by  the  decision  maker  to  compare  alternatives  and  to  select  a 
preferred  alternative. 

Source:  GAO-16-22  1  GAO-16-853 

1 .  Initialize  the  AOA  process:  includes  best  practices  that  are  applied 
before  starting  the  process  of  identifying,  analyzing,  and  selecting 
alternatives.  This  includes  determining  the  mission  need  and 
functional  requirements,  developing  the  study  time  frame,  creating  a 
study  plan,  and  determining  who  conducts  the  analysis. 

2.  Identify  alternatives:  includes  best  practices  that  help  ensure  the 
alternatives  to  be  analyzed  are  sufficient,  diverse,  and  viable. 

3.  Analyze  alternatives:  includes  best  practices  that  compare  the 
alternatives  to  be  analyzed.  The  best  practices  in  this  category  help 
ensure  that  the  team  conducting  the  analysis  uses  a  standard, 
quantitative  process  to  assess  the  alternatives. 

4.  Document  and  review  the  AOA  process:  includes  best  practices  that 
would  be  applied  throughout  the  AOA  process,  such  as  documenting 
all  steps  taken  to  initialize,  identify,  and  analyze  alternatives  and  to 
select  a  preferred  alternative  in  a  single  document. 

5.  Select  a  preferred  alternative:  includes  a  best  practice  that  is  applied 
by  the  decision  maker  to  compare  alternatives  and  to  select  a 
preferred  alternative. 

Also,  we  have  grouped  the  22  best  practices  into  four  characteristics  that 
identify  a  high-quality,  reliable  AOA  process — specifically,  that  it  be  well- 
documented,  comprehensive,  unbiased,  and  credible.  In  appendix  III,  we 
provide  more  information  on  these  best  practices,  and  below  in  table  2, 
we  show  how  each  of  them  are  grouped  into  the  four  characteristics. 
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Table  2:  GAO’s  22  Best  Practices  for  Analysis  of  Alternatives  (AOA)  Grouped  into 
Four  Characteristics 

Characteristics 

AOA  best  practices 

Well  documented:  The  AOA 
process  is  thoroughly  described, 
including  all  source  data,  clearly 
detailed  methodologies, 
calculations  and  results,  and 
selection  criteria  are  explained. 

12.  Identify  significant  risks  and  mitigation  strategies 
14.  Tie  benefits/effectiveness  to  mission  need 

18.  Document  AOA  process  in  a  single  document 

19.  Document  assumptions  and  constraints 

Comprehensive:  The  level  of 
detail  for  the  AOA  process 
ensures  no  alternatives  are 
omitted  and  that  each  alternative 
is  examined  thoroughly  for  the 
project’s  entire  life-cycle. 

I.  Define  mission  need 

3.  Develop  AOA  timeframe 

8.  Develop  list  of  alternatives 

I I .  Assess  alternatives’  viability 

15.  Develop  life-cycle  cost  estimates  (LCCEs) 

Unbiased:  The  AOA  process  does 
not  have  a  predisposition  towards 
one  alternative  over  another  but  is 
based  on  traceable  and  verified 
information 

2.  Define  functional  requirements 

4.  Establish  AOA  team 

6.  Weight  selection  criteria 

7.  Develop  AOA  process  plan 

13.  Determine  and  quantify  benefits  and 
effectiveness 

20.  Ensure  AOA  process  is  impartial 

22.  Compare  alternatives 

Credible:  The  AOA  process 
discusses  any  limitations  of  the 
analysis  resulting  from  the 
uncertainty  surrounding  the  data 
to  assumptions  made  for  each 
alternative. 

5.  Define  selection  criteria 

9.  Describe  alternatives 

10.  Include  baseline  alternative 

16.  Include  a  confidence  interval  or  range  for  LCCEs 

17.  Perform  sensitivity  analysis 

21.  Perform  independent  review 

Source:  GAO-16-22  1  GAO-16-853 

Further,  the  principles  and  practices  contained  in  GAO’s  AOA  best 
practices  parallel  those  found  in  DOD  and  Air  Force  guidance  on  military 
construction  and  analysis  for  decision  making.  For  example,  according  to 
an  Air  Force  instruction  governing  the  planning  and  programming  for 
military  construction  projects,  one  of  the  required  planning  actions  is  to 
evaluate  alternative  solutions.29  This  guidance  on  planning  clearly  agrees 
with  the  intent  of  our  AOA  best  practices,  in  which  we  state  that  the  best 

29Air  Force  Instruction  32-1 021 ,  Planning  and  Programming  Military  Construction 
(MILCON)  Projects,  (Feb.  25,  2016). 
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practices  can  be  applied  to  a  wide  range  of  activities  in  which  an 
alternative  must  be  selected  from  a  set  of  possible  options.  The 
congruence  between  our  AOA  best  practices  and  the  department’s 
guidance  on  these  subjects  demonstrates  our  best  practices’  applicability 
to  a  military  construction  project  such  as  JIAC  consolidation. 


DOD’s  AOA  Process 
Partially  Met  the  Four 
Characteristics  of  a  High- 
Quality,  Reliable  AOA 
Process 


By  comparing  DOD’s  AOA  process  to  our  best  practices,  we  determined 
that  DOD’s  process  partially  met  the  four  characteristics  of  a  high-quality, 
reliable  AOA  process.  In  table  3,  we  summarize  our  assessment. 


Table  3:  Summary  of  GAO’s  22  Analysis  of  Alternatives  (AOA)  Best  Practices,  Grouped  into  Four  Characteristics,  with  GAO’s 
Scores  for  DOD’s  AOA  Process  for  JIAC  Consolidation 

AOA  characteristic 

Overall  GAO  assessment3 

Well  documented:  The  analysis  of  alternatives  (AOA)  process  is  thoroughly  described,  including 
all  source  data,  clearly  detailed  methodologies,  calculations  and  results,  and  selection  criteria  are 
explained. 

Average  Score:  2.75  out  of  5 
Partially  met 

Comprehensive:  The  level  of  detail  for  the  AOA  process  ensures  no  alternatives  are  omitted  and 
that  each  alternative  is  examined  thoroughly  for  the  project’s  entire  life-cycle. 

Average  Score:  3.2  out  of  5 
Partially  met 

Unbiased:  The  AOA  process  does  not  have  a  predisposition  towards  one  alternative  over  another 
but  is  based  on  traceable  and  verified  information 

Average  Score:  3.0  out  of  5 
Partially  Met 

Credible:  The  AOA  process  discusses  any  limitations  of  the  analysis  resulting  from  the  uncertainty 
surrounding  the  data  to  assumptions  made  for  each  alternative. 

Average  Score:  3.0  out  of  5 
Partially  Met 

Source:  GAO  analysis  of  Department  of  Defense  (DOD)  information.  I  GAO-16-853 


aln  this  report,  we  use  a  five-point  scale  to  describe  our  assessment  of  DOD’s  AOA  process.  On  this 
scale,  each  of  the  22  best  practices  and  four  characteristics  is  scored  as  not  met,  minimally  met, 
partially  met,  substantially  met,  or  fully  met.  We  set  the  following  numerical  scores  for  each  step  on 
the  scale:  Not  Met  =  1 .0  to  1.4;  Minimally  Met  =  1 .5  to  2.4:  Partially  Met  =  2.5  to  3.4:  Substantially  Met 
=  3.5  to  4.4;  and  Fully  Met  =  4.5  to  5.0. 


Specifically,  DOD’s  AOA  process  for  JIAC  consolidation  fully  or 
substantially  met  6  best  practices,  partially  or  minimally  met  15  best 
practices,  and  did  not  meet  1  best  practice.  In  table  4,  we  summarize  our 
assessment.  Below  the  table,  we  highlight  certain  best  practices, 
providing  examples  from  our  analysis  for  each  of  the  four  characteristics. 
See  appendix  IV  for  a  summary  of  our  analysis  for  all  22  best  practices. 
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Table  4:  GAO’s  22  Analysis  of  Alternatives  (AOA)  Best  Practices,  Grouped  into  Four  Characteristics,  with  GAO’s  Scores  for 
DOD’s  AOA  Process 

Characteristics 

AOA  best  practices 

GAO  scoring  of 

DOD  AOA  process 

Well  documented:  The  analysis  of  alternatives 
(AOA)  process  is  thoroughly  described, 
including  all  source  data,  clearly  detailed 
methodologies,  calculations  and  results,  and 
selection  criteria  are  explained. 

12.  Identify  significant  risks  and  mitigation  strategies 

14.  Tie  benefits/effectiveness  to  mission  need 

18.  Document  AOA  process  in  a  single  document 

19.  Document  assumptions  and  constraints 

3  -  Partially  Met 

3  -  Partially  Met 

2  -  Minimally  Met 

3  -  Partially  Met 

Overall  GAO  Assessment3  Partially  Met 

Average  Score:  2.75  out  of  5 

Comprehensive:  The  level  of  detail  for  the 

AOA  process  ensures  no  alternatives  are 
omitted  and  that  each  alternative  is  examined 
thoroughly  for  the  project’s  entire  life-cycle. 

I.  Define  mission  need 

3.  Develop  AOA  timeframe 

8.  Develop  list  of  alternatives 

I I .  Assess  alternatives’  viability 

15.  Develop  life-cycle  cost  estimates  (LCCEs) 

5  -  Fully  Met 

3  -  Partially  Met 

4  -  Substantially  Met 

2  -  Minimally  Met 

2  -  Minimally  Met 

Overall  GAO  Assessment3  Partially  Met 

Average  Score:  3.2  out  of  5 

Unbiased:  The  AOA  process  does  not  have  a 
predisposition  towards  one  alternative  over 
another  but  is  based  on  traceable  and  verified 
information 

2.  Define  functional  requirements 

4.  Establish  AOA  team 

6.  Weight  selection  criteria 

7.  Develop  AOA  process  plan 

13.  Determine  and  quantify  benefits  and 
effectiveness 

20.  Ensure  AOA  process  is  impartial 

22.  Compare  alternatives 

5  -  Fully  Met 

3  -  Partially  Met 

3  -  Partially  Met 

2  -  Minimally  Met 

2  -  Minimally  Met 

4  -  Substantially  Met 

2  -  Minimally  Met 

Overall  GAO  Assessment3  Partially  Met 

Average  Score:  3.0  out  of  5 

Credible:  The  AOA  process  discusses  any 
limitations  of  the  analysis  resulting  from  the 
uncertainty  surrounding  the  data  to 
assumptions  made  for  each  alternative. 

5.  Define  selection  criteria 

9.  Describe  alternatives 

10.  Include  baseline  alternative 

16.  Include  a  confidence  interval  or  range  for  LCCEs 

17.  Perform  sensitivity  analysis 

21.  Perform  independent  review 

5  -  Fully  Met 

2  -  Minimally  Met 

4  -  Substantially  Met 

1  -  Not  Met 

3  -  Partially  Met 

3  -  Partially  Met 

Overall  GAO  Assessment3  Partially  Met 

Average  Score:  3.0  out  of  5 

Source:  GAO  analysis  of  Department  of  Defense  (DOD)  information.  I  GAO-16-853 


aWe  determined  the  overall  assessment  rating  by  assigning  each  individual  rating  a  number:  Not  Met 
=  1 .  Minimally  Met  =  2,  Partially  Met  =3,  Substantially  Met  =  4,  and  Fully  Met  =  5.  Then,  we  took  the 
average  of  the  individual  assessment  ratings  to  determine  the  overall  rating  for  each  of  the  four 
characteristics.  The  resulting  average  becomes  the  Overall  Assessment  as  follows:  Not  Met  =  1.0  to 
1 .4,  Minimally  Met  =  1 .5  to  2.4,  Partially  Met  =  2.5  to  3.4,  Substantially  Met  =  3.5  to  4.4,  and  Fully  Met 
=  4.5  to  5.0. 
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DOD’s  AOA  Team  Partially 
Met  the  Characteristic  of  a  Well 
Documented  AOA  Process 


Overall,  DOD’s  AOA  team  partially  met  the  collection  of  best  practices 
that  address  how  well  an  agency  documented  its  AOA  process.  For 
example,  DOD’s  AOA  process  partially  met  the  best  practice  of  Tying 
Benefits  or  Effectiveness  to  Mission  Need  and  minimally  met  the  best 
practice  of  Documenting  the  AOA  Process  in  a  Single  Document. 

The  requirements  of  this  best  practice — Tying  Benefits  or  Effectiveness  to 
Mission  Need — include,  among  other  things,  that  the  AOA  team  explain 
how  each  measure  of  effectiveness  supports  the  mission  need,  and  show 
how  the  measures  are  tied  to  specific  mission  needs  and  functional 
requirements.  DOD’s  AOA  body  of  work  includes  the  201 1  preliminary 
AOA  summary  in  which  the  AOA  team  provides  a  general  explanation  of 
how  mission  needs  will  or  will  not  be  met  under  each  of  DOD’s  AOA 
criteria.  For  example,  in  the  context  of  the  criterion  “Impact  on  Bilateral 
and  Multinational  Intelligence  Collaboration,”  DOD’s  2011  preliminary 
AOA  summary  explains  that  the  JIAC’s  location  plays  a  key  role  in  the 
complex’s  ability  to  leverage  partner  nation  intelligence  capabilities,  such 
as  those  of  the  United  Kingdom,  and  its  proximity  to  the  NATO 
Intelligence  Fusion  Center  gives  the  JIAC  ready  access  to  the  individual 
and  fused  intelligence  capabilities  of  the  center’s  26  participating  nations. 
This  discussion  indicates  that,  generally  speaking,  location  of  the  JIAC  is 
tied  to  its  mission  effectiveness.  However,  the  AOA  body  of  work  does  not 
provide  evidence  that  the  AOA  team  developed  specific  measures  of 
effectiveness — used  to  quantify  the  benefits  and  effectiveness  of  each 
alternative — or  tied  those  measures  to  specific  mission  needs.  According 
to  the  actions  required  under  the  best  practice  Determine  and  Quantify 
Benefits/Effectiveness — on  which  the  best  practice  Tying  Benefits  or 
Effectiveness  to  Mission  Need  is  based — there  are  ways  in  which  an 
agency  can  meet  this  best  practice  with  largely  qualitative  criteria,  like 
those  used  by  the  AOA  team.  Even  in  cases  where  it  may  be  difficult  to 
quantify  benefits,  an  AOA  team  can  either  explain  why  benefits  cannot  be 
quantified  or  use  scalability  assessments  to  quantify  benefits.  For 
instance,  for  the  criterion  “Impact  on  Bilateral  and  Multinational 
Intelligence  Collaboration,”  the  AOA  team  could  have  estimated  the 
number  of  face-to-face  interactions  between  U.S.  and  United  Kingdom 
intelligence  personnel  made  possible  by  each  alternative  and  used  that 
as  a  metric  to  consistently  compare  alternatives.  Or,  the  team  could  have 
provided  evidence  that  it  assessed  other  mechanisms  for  collaborating 
with  partner  nation  personnel,  in  contrast  with  face-to-face  collaboration. 
This  type  of  assessment  might  have  provided  evidence  that  strengthens 
or  weakens  DOD’s  case  for  prioritizing  face-to-face  collaboration. 
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DOD’s  AOA  Team  Partially 
Met  the  Characteristic  of  a 
Comprehensive  AOA  Process 


In  addition,  DOD’s  AOA  process  minimally  met  the  best  practice  of 
Documenting  the  AOA  Process  in  a  Single  Document.  This  best  practice 
requires,  among  other  things,  that  the  AOA  team  documents  all  steps 
taken  to  identify,  analyze,  and  select  alternatives  in  a  single  document. 
DOD’s  AOA  body  of  work  contains  one  summary  document:  the  201 1 
preliminary  AOA  summary.  However,  the  201 1  preliminary  AOA  summary 
does  not  contain  all  of  the  information  in  DOD’s  AOA  body  of  work.  Also, 
according  to  DOD  officials,  the  AOA  team  produced  the  document  in 
August  201 1 ,  more  than  one  and  one-half  years  before  the  end  of  DOD’s 
AOA  process  in  April  2013.  Further,  the  officials  stated  that  there  is  no 
other  document  that  the  department  considers  to  be  an  official  summary 
of  the  AOA  process.  According  to  our  best  practices,  documentation  is 
essential  for  validating  and  defending  the  AOA  process.  Without  clear 
reports  that  compile  all  information,  including  standards  used  to  rate  and 
perform  the  analysis,  the  AOA’s  credibility  could  suffer  because  the 
documentation  does  not  explain  the  rationale  for  the  methodology  used  or 
the  calculations  underlying  the  analysis. 

Overall,  DOD’s  AOA  team  partially  met  the  collection  of  best  practices 
that  demonstrate  the  comprehensiveness  of  an  agency’s  AOA  process. 
For  example,  DOD’s  AOA  process  fully  met  the  best  practice  of  Defining 
Mission  Need  and  substantially  met  the  best  practice  of  Developing  a  List 
of  Alternatives.  However,  DOD’s  AOA  process  minimally  met  the  best 
practices  of  Assessing  Alternatives’  Viability  and  Developing  Life-Cycle 
Cost  Estimates. 

The  requirements  of  the  best  practice  Defining  Mission  Need  include, 
among  other  things,  that  the  AOA  team  define  the  mission  needs — i.e.,  a 
credible  gap  between  current  capabilities  and  those  required  to  meet 
goals — without  a  predetermined  solution.  The  department’s  AOA  body  of 
work  included  DOD  1391  forms — used  to  provide  Congress  with 
information  on  military  construction  projects — that  include  information  on 
the  gap  between  current  JIAC  facilities’  capabilities  and  those  required  to 
meet  JIAC  mission  needs.  Examples  of  issues  listed  on  the  1391  forms 
include  a  severe  shortfall  of  intelligence  spaces,  resulting  in  missions 
being  housed  in  undersized  facilities,  and  aging  and  inefficient  power  and 
cooling  systems  critical  to  the  intelligence  mission  that  are  not  able  to  be 
economically  upgraded.  The  forms  explain  that  these  facility  shortfalls  will 
continue  to  constrain  the  JIAC’s  ability  to  provide  responsive  intelligence 
in  support  of  EUCOM  and  AFRICOM.  The  AOA  body  of  work  also 
included  detailed  requirements  documents  that  laid  out  both  the  shortfalls 
of  current  JIAC  facilities  and  the  specifications  for  a  new  JIAC  facility, 
based  on  the  mission  needs  of  the  JIAC’s  organizations.  Further,  these 
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documents  state  that  DOD  had  not  yet  determined  where  to  consolidate 
the  JIAC.  According  to  this  best  practice,  allowing  mission  needs  to  be 
defined  in  solution-specific  terms  creates  a  potential  bias  and  could 
invalidate  the  analysis.  In  fully  meeting  the  best  practice,  the  AOA  team 
demonstrated  that  this  initial  component  of  its  analysis  was  free  of  bias. 

Also,  DOD’s  AOA  process  substantially  met  the  best  practice  of 
Developing  a  List  of  Alternatives.  This  best  practice  requires,  among 
other  things,  that  the  AOA  team  identifies  and  considers  a  wide  range  of 
alternatives  to  meet  the  mission  need  and  perform  market  research  to 
develop  as  many  alternative  solutions  as  possible.  According  to  DOD 
officials,  the  department  considered  up  to  16  alternatives.  This  indicates 
that  the  AOA  team  identified  and  considered  a  wide  range  of  alternatives 
to  meet  the  mission  need.  Regarding  the  requirement  to  perform  market 
research,  it  seems  reasonable  that  the  alternatives  developed  by  the  AOA 
team  were  largely  European,  given  that  three  of  DOD’s  five  AOA 
criteria — and  both  critical  criteria — were  focused  on  intelligence 
collaboration  and  relationships  with  European  partner  nations.  According 
to  our  best  practice,  an  AOA  process  like  DOD’s — that  encompasses 
numerous  alternatives — ensures  the  study  provides  a  broad  view  of  the 
issue.  However,  the  AOA  body  of  work  did  not  contain  evidence  that  the 
AOA  team  performed  other  types  of  market  research  that  may  have 
informed  the  AOA  process.  For  example,  given  that  some  of  the  JIAC’s 
military,  civilian,  and  contractor  employees — and  their  dependents — will 
live  off  base,  the  AOA  team  may  have  benefited  from  performing  research 
on  the  commercial  housing  markets  adjacent  to  the  alternatives. 

However,  DOD’s  AOA  process  minimally  met  the  best  practice  of 
Assessing  Alternatives’  Viability.  This  best  practice’s  requirements  include 
the  AOA  team’s  screening  of  the  list  of  alternatives,  in  order  to  eliminate 
those  alternatives  that  are  not  viable,  and  fully  documenting  all 
assumptions  regarding  the  alternatives’  viable  and  nonviable  status — 
including  reasons  that  an  alternative  is  not  viable — in  order  to  justify  the 
AOA  process’  eventual  recommendation.  According  to  DOD  officials,  the 
AOA  team  determined  alternatives’  viability  by  examining  the  alternatives 
using  the  department’s  own  five  AOA  criteria  discussed  above.  However, 
the  DOD  AOA  body  of  work  does  not  contain  documentation  indicating 
that  the  AOA  team  met  several  requirements  of  our  best  practice.  For 
example,  the  body  of  work  does  not  contain  a  documented  description  of 
the  process  that  the  AOA  team  used  to  add  and  remove  alternatives  over 
time — 16  in  total — as  DOD  progressed  through  its  AOA  process.  DOD 
officials  confirmed  that  such  a  document  does  not  exist  and  were  unable 
to  explain  how  each  alternative  was  eliminated  from  consideration.  Also, 
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there  is  no  documentation  of  the  reasons  why  15  of  these  alternatives 
were  eventually  considered  not  to  be  viable.  According  to  this  best 
practice,  documenting  the  alternatives  that  are  deemed  nonviable  is 
important  so  that  decision  makers  can  clearly  see  why  those  alternatives 
are  not  considered  for  further  analysis.  Without  such  documentation, 

DOD  is  less  able  to  justify  the  AOA  process’  eventual  recommendation. 

Further,  DOD’s  AOA  process  minimally  met  the  best  practice  of 
Developing  Life-Cycle  Cost  Estimates  during  an  AOA  process.  The 
requirements  of  this  best  practice  include,  among  other  things,  that  the 
AOA  team  develop  a  life-cycle  cost  estimate  for  each  alternative.  This  life- 
cycle  cost  estimate  should  include  all  costs  from  inception  of  the  effort — in 
this  case,  JIAC  consolidation — through  design,  development, 
construction,  operation,  maintenance,  and  disposal.  As  discussed  above, 
while  DOD  focused  on  non-financial  criteria  in  the  AOA  process,  the 
department  included  consideration  of  alternatives’  costs  as  part  of  its 
AOA  process,  under  its  AOA  criterion  “Business  Case.”  DOD’s  AOA  body 
of  work  provides  evidence  that  the  AOA  team  developed  partial  life  cycle 
cost  estimates  as  part  of  the  AOA  process.  For  example,  the  201 1 
preliminary  AOA  summary  shows  the  team  considered  life  cycle  costs 
when  selecting  alternatives  for  analysis,  and  a  table  in  the  document 
provides  both  military  construction  costs  and  projected  cost  savings  or 
cost  avoidance  that  are  associated  with  each  alternative. 

However,  the  documentation  in  DOD’s  AOA  body  of  work  does  not 
provide  the  information  required  for  full  life  cycle  cost  estimates.  Officials 
from  the  Office  of  the  Secretary  of  Defense  and  Air  Force  Headquarters 
characterized  the  level  of  detail  for  these  estimates  as  “extremely  rough.” 
Officials  from  these  organizations  agree  that  these  estimates  are  not  full 
life  cycle  cost  estimates.  For  instance,  the  estimates  in  the  201 1 
preliminary  AOA  summary’s  table  do  not  include  all  of  the  types  of  costs 
that  make  up  the  JIAC  consolidation  effort.  According  to  our  review  of  the 
table,  it  provides  no  evidence  that  its  estimates  include  costs  such  as 
additional  housing,  personnel  costs,  support  personnel,  or  the  base 
operations  support,  which  according  to  JIAC  officials  will  be  needed  to 
accommodate  the  increased  logistical  requirements  the  JIAC  and  its 
personnel  would  place  on  RAF  Croughton’s  infrastructure.  Also,  the  table 
does  not  provide  evidence  that  its  estimates  include  all  savings  that  DOD 
has  estimated  could  be  achieved  through  JIAC  consolidation.  Specifically, 
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DOD’s  AOA  Team  Partially 
Met  the  Characteristic  of  an 
Unbiased  AOA  Process 


while  the  table  includes  some  estimated  savings,  the  table  does  not 
include  other  potential  savings.30  For  example,  DOD  officials  have 
explained  that  the  RAF  Croughton  alternative  assumes  closure  of  RAF 
Molesworth  and  RAF  Alconbury,  and  according  to  EUCOM 
documentation  and  officials  from  the  Office  of  the  Secretary  of  Defense 
and  Air  Force  headquarters,  there  will  be  a  reduced  number  of  personnel 
required  to  meet  requirements  for  one  installation  (RAF  Croughton)  as 
opposed  to  two  (RAF  Molesworth  and  RAF  Alconbury).  However,  these 
savings  from  this  consolidation  alternative  were  not  included  in  the 
table.31  Further,  DOD’s  AOA  body  of  work  has  no  documentation  of  the 
data  or  analysis  that  was  used  to  produce  the  cost  estimates  found  in  the 
201 1  preliminary  AOA  summary  and  DOD  officials  were  unable  to  provide 
these  data.  According  to  this  best  practice,  an  incomplete  life-cycle  cost 
estimate  does  not  provide  an  accurate  and  complete  view  of  the 
alternatives’  costs.  Without  a  full  accounting  of  life-cycle  costs,  decision 
makers  will  not  have  a  complete  picture  of  the  costs  for  each  alternative 
and  will  have  difficulty  comparing  the  alternatives.  This  may  be 
particularly  true  of  DOD’s  AOA  process  for  JIAC  consolidation  because, 
as  discussed  above,  the  relative  importance  of  cost  in  the  AOA  team’s 
analysis  is  somewhat  unclear. 

Overall,  DOD’s  AOA  team  partially  met  the  collection  of  best  practices 
that  demonstrate  the  extent  to  which  an  agency  conducted  its  AOA 
process  in  an  unbiased  fashion.  For  example,  DOD’s  AOA  process 
substantially  met  the  best  practice  of  Ensuring  that  the  AOA  Process  is 
Impartial  but  minimally  met  the  best  practice  of  Comparing  Alternatives. 

There  are  a  number  of  requirements  for  the  best  practice  Ensuring  that 
the  AOA  Process  is  Impartial.  These  requirements  include  that  the  AOA 
team  conduct  the  analysis  without  a  predetermined  solution  and  that  the 
AOA  process  informs  the  decision-making  process  rather  than  reflecting 
the  validation  of  a  predetermined  solution.  Although  the  AOA  team 
developed  the  recommendation  to  consolidate  the  JIAC  at  RAF 
Croughton,  this  recommendation  was  not  considered  to  be  DOD’s  final 


30The  table  includes  estimated  annual  savings,  in  military  construction  costs,  compared  to 
the  status  quo  (defined  as  deferring  consolidation  of  the  JIAC). 

31These  estimated  savings  will  be  discussed  in  more  detail  in  a  separate  review  focused 
on  DOD’s  cost  estimate  for  consolidation  of  the  JIAC  at  RAF  Croughton. 
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decision  until  the  end  of  the  AOA  process.  Key  documents  in  the  AOA 
body  work  provide  evidence  that  the  AOA  team  conducted  its  analysis 
without  a  predetermined  solution.  For  example,  neither  a  briefing  for 
EUCOM  leadership  nor  the  201 1  preliminary  AOA  summary  refer  to  the 
recommendation  of  consolidation  at  RAF  Croughton  as  the  final  choice  of 
the  AOA  team.  Also,  as  discussed  above,  the  JIAC  requirements 
document  states  that  DOD  had  not  yet  determined  where  to  consolidate 
the  JIAC.  In  addition,  according  to  officials  from  the  Office  of  the 
Secretary  of  Defense  and  Air  Force  headquarters,  a  DOD  organization 
independent  of  the  AOA  team  reviewed  some  of  the  AOA  team’s 
conclusions.  The  officials  noted  that  the  Office  of  Cost  Assessment  & 
Program  Evaluation — which  reports  directly  to  the  Secretary  of  Defense 
and  is  responsible  for  managing  DOD’s  process  for  making  project 
funding  decisions — reviewed  certain  documentation  related  to  the 
proposal  to  fund  JIAC  consolidation  at  RAF  Croughton.  As  discussed 
previously,  DOD’s  body  of  work  does  not  contain  a  documented 
description  of  the  process  that  the  AOA  team  used  to  add  or  remove 
alternatives  as  the  department  progressed  through  the  AOA  process. 
Therefore,  we  cannot  fully  determine  the  impartiality  of  the  process. 
However,  by  substantially  meeting  this  best  practice,  DOD’s  AOA  team 
provided  reasonable  assurance  to  decision  makers  that  the  AOA  process 
for  JIAC  consolidation  can  be  considered  to  be  valid  and  un-biased. 

In  addition,  DOD’s  AOA  process  minimally  met  the  best  practice  of 
Comparing  Alternatives.  The  requirements  of  this  best  practice  include, 
among  other  things,  that  the  AOA  team — or  another  entity  in  the  AOA 
process — compare  the  alternatives  using  net  present  value  to  select  a 
preferred  alternative,  if  possible.32  DOD’s  AOA  body  of  work  does  not 
provide  evidence  that  the  AOA  team  used  net  present  value  to  compare 
alternatives  and  officials  from  the  Office  of  the  Secretary  of  Defense 
confirmed  that  type  of  comparison  was  not  performed  during  the  AOA 
process.  Instead,  DOD’s  body  of  work  provides  evidence  that  the  AOA 
team  used  two  other  approaches  to  compare  most  of  the  alternatives  to 


32Net  present  value  is  a  tool  that  can  be  used  to  appraise  multi-year  projects  and  is  the 
present  value  of  expected  benefits  minus  the  present  value  of  expected  costs.  The 
present  value  of  a  stream  of  future  benefits  or  costs  is  its  worth  in  terms  of  money  paid 
today.  Present  value  calculations  reflect  the  time  value  of  money,  based  on  the 
assumption  that  a  dollar  in  the  future  is  worth  less  than  a  dollar  today  because  the  dollar 
today  can  be  invested  and  earn  interest. 
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each  other  against  the  department’s  five  AOA  criteria.  Specifically,  the 
201 1  preliminary  AOA  summary  contains  two  tables,  comparing 
alternatives  according  to  these  AOA  criteria,  as  well  as  by  certain  types  of 
costs.  These  tables  allow  an  outside  observer  to  gain  some  insight  into 
the  alternatives’  relative  scores.  However,  the  table  on  alternatives  does 
not  contain  each  of  the  1 1  alternatives  listed  in  the  document  and  the 
table  on  cost  does  not  contain  a  full  set  of  costs  for  each  alternative. 
According  to  the  best  practice,  it  may  be  appropriate  in  certain  cases  to 
not  use  net  present  value,  such  as  when  quantifying  benefits  is  not 
possible.  In  cases  such  as  DOD’s  AOA  process  for  JIAC  consolidation — 
where  the  AOA  team  did  not  use  net  present  value — the  team  should  take 
certain  steps,  like  documenting  why  net  present  value  was  not  used, 
describe  the  other  method  that  is  used  to  differentiate  between 
alternatives,  and  explain  why  that  method  has  been  applied.  DOD’s  AOA 
body  of  work  does  not  provide  evidence  that  the  AOA  team  took  these 
steps.  Because  the  AOA  team  did  not  use  net  present  value,  the 
comparison  method  used  by  the  team  did  not  include  all  alternatives  or  all 
of  their  costs,  and  the  team  did  not  explain  these  issues,  DOD  is  likely  to 
have  difficulty  explaining  how  the  AOA  team’s  comparison  resulted  in 
RAF  Croughton  as  the  preferred  alternative. 

Overall,  DOD’s  AOA  team  partially  met  the  collection  of  best  practices 
that  demonstrate  the  credibility  of  an  agency’s  AOA  process.  For 
example,  DOD’s  AOA  process  fully  met  the  best  practice  of  Defining 
Selection  Criteria  but  minimally  met  the  best  practice  of  Describing 
Alternatives. 

Requirements  for  the  best  practice  of  Defining  Selection  Criteria  include 
defining  the  selection  criteria  based  on  mission  needs  and  ensuring  that 
the  criteria  are  independent  of  a  particular  capital  asset  or  technological 
solution.  DOD’s  body  of  work  shows  that  the  AOA  team  considered  the 
JIAC’s  mission  needs  when  developing  the  AOA  criteria.  For  example, 
the  201 1  preliminary  AOA  summary  defines  the  second  criterion — ’’Impact 
on  Bilateral  and  Multinational  Intelligence  Collaboration” —  in  the  context 
of  the  JIAC’s  mission,  explaining  that  given  the  important  role 
collaboration  plays  in  its  operations,  options  were  assessed  for  their 
potential  impacts  on  international  intelligence  partnerships.  In  addition, 
DOD’s  criteria  are  independent  of  a  particular  solution  to  the  requirements 
that  the  AOA  team  was  trying  to  meet.  For  instance,  while  the  201 1 
preliminary  AOA  summary’s  explanation  of  the  criterion  “Impact  on 
Intelligence  Operations”  states  that  the  JIAC’s  location  has  the  potential 
to  impact  certain  JIAC  operations,  this  explanation  does  not  use  a 
particular  location  to  define  what  the  criterion  means.  According  to  this 


Page  32 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


best  practice,  it  is  essential  that  the  selection  criteria  be  based  on  the 
mission  needs.  By  fully  meeting  this  best  practice,  the  AOA  team 
provided  confidence  that  bias  did  not  enter  into  this  part  of  DOD’s  AOA 
process  for  JIAC  consolidation. 

However,  DOD’s  AOA  process  minimally  met  the  best  practice  of 
Describing  Alternatives.  Several  of  our  best  practices  assess  how  an 
AOA  team  includes  alternatives  in  its  AOA  process.  For  one  of  these  best 
practices — Including  a  Baseline  Alternative — the  AOA  team  needs  to, 
among  other  things,  include  one  alternative  to  represent  the  status  quo  in 
order  to  provide  a  basis  of  comparison  among  alternatives.  We  assessed 
that  DOD’s  AOA  team  substantially  met  the  best  practice.  For  a  summary 
of  our  analysis  for  this  best  practice,  see  appendix  IV.  However,  DOD’s 
AOA  body  of  work  does  not  provide  evidence  that  the  AOA  team  met  the 
requirements  for  the  best  practice  of  Describing  Alternatives,  for  which 
the  AOA  must  provide  a  detailed  description  of  all  alternatives. 
Specifically,  requirements  for  this  best  practice  include  that  the  AOA  team 
describes  alternatives  in  sufficient  detail  to  allow  for  robust  analysis  and 
that  this  description  is  detailed  enough  to  support  the  AOA  team’s 
viability,  cost,  and  benefit/effectiveness  analyses.  However,  in  the  2011 
preliminary  AOA  summary,  the  AOA  team  described  the  alternatives  in 
basic  terms,  using  several  words  or  one  sentence.  Further,  the  DOD  body 
of  work  does  not  provide  sufficient  detail  on  each  alternative  to  allow  for 
robust  analysis.  Specifically,  the  alternatives’  descriptions  do  not  address 
their  ability  to  meet  JIAC  operational  requirements.  For  example,  in  the 
201 1  preliminary  AOA  summary,  the  complete  descriptions  provided  for 
the  two  alternatives — RAF  Alconbury  and  U.S.  Army  Garrison  Benelux — 
are  “construct  new  facilities  at  RAF  Alconbury”  and  “construct  new 
facilities  at  U.S.  Army  Garrison  Benelux,”  respectively.  This  level  of  detail 
does  not  allow  us  to  assess  the  difference  in  how  these  two  alternatives 
would  differ  in  meeting — for  example — DOD’s  third  criterion  “Impact  on 
International  Agreements  and  Relationships.”  Because  the  AOA  team  did 
not  adequately  describe  or  document  the  alternatives,  the  team’s  analysis 
did  not  provide  sufficient  detail  and  may  not  be  credible. 
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DOD  Officials  Did  Not  Use 
Best  Practices  to  Guide  the 
JIAC  Consolidation  AOA 
Process 


DOD  officials  stated  that  the  AOA  team  did  not  use  best  practices  to 
guide  the  JIAC  consolidation  AOA  process.  According  to  an  official  from 
the  Office  of  the  Secretary  of  Defense,  DOD  organizations  generally 
follow  a  standard  process  for  developing  decision  briefings,  like  some  of 
the  briefings  provided  to  DOD  senior  leaders  during  the  AOA  process. 
This  process  includes — among  other  things — developing  screening 
criteria,  comparing  courses  of  action,  and  making  a  recommendation.  The 
official  explained  that  DOD’s  AOA  team  generally  followed  this  approach 
when  conducting  the  AOA  for  JIAC  consolidation.  At  a  high  level,  this 
DOD  process  for  developing  decision  briefings  aligns  with  the  process 
described  in  our  AOA  best  practices.33  The  high-level  alignment 
notwithstanding,  DOD  did  not  fully  follow  best  practices  in  its  AOA 
process.  This  is  because — according  to  officials  from  the  Office  of  the 
Secretary  of  Defense  and  Air  Force  Headquarters — DOD  does  not  have  a 
set  of  best  practices  for  conducting  analysis  of  alternatives  for  military 
construction  project  decisions.  Further,  officials  from  the  Office  of  the 
Secretary  of  Defense,  EUCOM,  and  Air  Force  headquarters  explained 
that  members  of  the  AOA  team  did  not  follow  a  single,  particular  piece  of 


oo 

For  example,  both  processes  discuss  describing  the  background  and  assumptions 
included  in  the  analysis,  screening  alternatives  for  viability  before  performing  analysis 
based  on  predetermined  criteria,  evaluating  the  remaining  alternatives,  and  selecting  a 
preferred  alternative.  However,  the  GAO  best  practices  provide  more  details  regarding  the 
process  needed  to  initialize  an  analysis  of  alternatives,  thorough  documentation,  the 
specifics  involved  with  analyzing  alternatives,  and  the  need  for  an  independent  review  of 
the  AOA  process  at  various  stages  in  its  development. 
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AOA  guidance  or  set  of  AOA  best  practices  during  the  AOA  process.34 
While  DOD’s  AOA  process  substantially  or  fully  met  several  best 
practices,  overall,  the  department  partially  met  each  of  the  four 
characteristics  of  a  high-quality,  reliable  AOA  process.  We  have  found 
that  not  conforming  to  the  best  practices  may  lead  to  an  unreliable  AOA, 
and  in  those  cases,  the  agency  will  not  be  positioned  to  ensure  that  the 
preferred  solution  best  meets  the  mission  needs.  In  addition,  not  fully 
conforming  to  best  practices  can  lead  to  oversight  challenges.  For 
example,  DOD’s  lack  of  full  documentation  may  make  it  difficult  for 
decision  makers  in  the  department  and  Congress  to  fully  assess  how 
DOD  arrived  at  its  final  decision  to  consolidate  the  JIAC  at  RAF 
Croughton.  Further,  the  consolidation  of  the  JIAC  at  RAF  Croughton  is 
one  of  26  initiatives  that  DOD  has  decided  to  carry  out  as  part  of  the 
department’s  European  Infrastructure  Consolidation  effort.  Congress’ 
oversight  of  similar  overseas  basing  efforts  is  likely  to  continue  as  DOD 
implements  these  26  initiatives.  Also,  not  fully  conforming  to  best 
practices  may  make  it  more  difficult  for  DOD  to  defend  its  choice  of  RAF 
Croughton.  For  instance,  when  describing  alternatives,  our  best  practices 
state — in  part — that  documentation  is  essential  for  validating  the  AOA 
process  and  defending  its  conclusions.  Similarly — according  to  our  best 
practice  of  documenting  the  AOA  process  in  a  single  document — without 
clear  reports  that  compile  all  information,  the  study’s  credibility  could 
suffer  because  the  documentation  does  not  explain  the  rationale  for  the 
methodology  or  the  calculations  underlying  the  analysis.  Officials  from  the 
Office  of  the  Secretary  of  Defense  and  Air  Force  headquarters 
acknowledged  that  DOD  does  not  have  the  level  of  documentation  that 
would  allow  GAO  to  recreate  the  AOA  team’s  analyses  and  conclusions. 


34The  Air  Force  has  an  AOA  Handbook:  Office  of  Aerospace  Studies,  U.S.  Air  Force 
Materiel  Command,  Analysis  of  Alternatives  (AOA)  Handbook:  A  Practical  Guide  to 
Analyses  of  Alternatives  (June  2013).  After  reviewing  the  Air  Force  AOA  guide,  we 
determined  that  the  Air  Force  guidance  has  language  that  directly  addresses  the  majority 
of  our  AOA  best  practices.  However,  there  are  certain  cases  in  which  the  Air  Force  AOA 
guidance  differs  from  our  AOA  best  practices.  For  example,  the  Air  Force  handbook  does 
not  recommend  the  use  of  weighting  schemes  as  part  of  the  AOA  effectiveness 
methodology.  This  is  in  contrast  to  our  AOA  best  practice  6,  which  calls  for  the  AOA  team 
to  weight  selection  criteria. 
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As  discussed  previously  in  this  report,  we  shared  our  draft  analysis  of 
DOD’s  AOA  process  with  the  department.  During  our  discussions,  DOD 
officials  stated  that  they  see  the  value  of  our  AOA  best  practices,  but  also 
provided  several  reasons  why  they  believe  the  best  practices  do  not  all 
apply  to  DOD’s  process  for  making  military  construction  decisions.  For 
example,  DOD  officials  explained  that  they  do  not  believe  they  needed  to 
fully  adhere  to  our  AOA  best  practices  during  their  AOA  process  because 
the  decision  to  fund  the  military  construction  project  for  consolidation  of 
the  JIAC  at  RAF  Croughton  was  considered  and  then  approved  through 
the  department’s  Program,  Budget,  and  Review  process.35  While  this  is 
the  case,  and  we  assessed  documentation  from  that  process  for  our 
review,  DOD’s  Program,  Budget,  and  Review  process  was  not  a 
substitute  for  conducting  a  robust  AOA  for  JIAC  consolidation.  This  is 
because — according  to  DOD  officials — DOD’s  Program,  Budget,  and 
Review  process  for  the  JIAC  consolidation  project  occurred  at  the  end  of 
DOD’s  AOA  process  and,  during  the  Program,  Budget,  and  Review 
process,  the  department  considered  only  the  preferred  solution  of  RAF 
Croughton.  Thus,  the  Program,  Budget  and  Review  process  cannot  serve 
as  a  substitute  for  an  AOA  process  that  adheres  to  all  of  the  AOA  best 
practices. 


Also,  according  to  department  officials,  the  professional  military  judgment 
of  senior  DOD  leaders  played  a  significant  role  in  their  decision  to  fund 
the  military  construction  project  for  consolidation  of  the  JIAC  at  RAF 
Croughton  and  thus  diminished  the  need  to  fully  adhere  to  our  AOA  best 
practices  during  the  AOA  process.  The  officials  with  whom  we  discussed 
our  preliminary  observations  explained  that  our  best  practices’  focus  on 
analyses  supporting  AOA  decision  making  do  not  fully  account  for  the  role 
of  professional  military  judgment  in  the  Program,  Budget  and  Review 
process.  In  the  case  of  JIAC  consolidation,  DOD  officials  explained  that 
RAF  Croughton  entered  the  Program,  Budget  and  Review  process  as  the 
preferred  solution  endorsed  by  the  commanders  of  both  EUCOM  and 
AFRICOM;  at  the  process’  conclusion,  the  Deputy  Secretary  of  Defense 


35According  to  DOD  officials,  the  Planning,  Programming,  and  Budgeting  process  in  which 
the  JIAC  consolidation  military  construction  project  was  considered  was  part  of  the 
department’s  Planning,  Programming,  Budgeting,  and  Execution  process.  According  to 
DOD  Directive  7045.14,  The  Planning ,  Programming ,  Budgeting,  and  Execution  (PPBE) 
Process  (Jan.  25,  2013),  this  process  serves  as  the  annual  resource  allocation  process  for 
DOD  within  its  4-year  planning  cycle. 
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approved  RAF  Croughton.  However,  our  AOA  best  practices  incorporate 
the  professional  judgment  of  senior  agency  officials.  Specifically, 
according  to  our  best  practices,  two  of  the  key  entities  involved  in  the 
AOA  process — the  customer  and  decision  maker — are  expected  to 
exercise  their  professional  judgment  in  making  key  decisions.36  For 
example,  in  our  best  practice  of  Defining  Mission  Need,  the  customer 
defines  the  gap  between  current  capabilities  and  those  required  to  meet 
the  agency’s  goals.  Also,  for  our  best  practice  Define  Selection  Criteria, 
the  AOA  team  or  the  decision  maker  defines  selection  criteria  based  on 
the  mission  need.  Our  assessment  that  the  AOA  team  fully  met  both  of 
these  best  practices — discussed  in  more  detail  later  in  this  report — is 
evidence  that  DOD  can  follow  these  AOA  best  practices  while  its  leaders 
exercise  their  professional  military  judgment. 

In  addition,  DOD  officials  stated  that  our  best  practices  seem  to  apply 
more  to  AOA  processes  for  acquisition  of  a  weapons  system  than  to  the 
department’s  decisions  about  military  construction.  However,  we  believe 
that  our  AOA  best  practices  can  be  applied  to  a  wide  range  of  activities  in 
which  an  alternative  must  be  selected  from  a  set  of  possible  options,  as 
well  as  to  a  broad  range  of  capability  areas,  projects,  and  programs — 
including  DOD’s  military  construction  decision  making  processes.  In 
developing  our  AOA  best  practices,  we  reviewed  a  variety  of  U.S. 
government  guidance,  including  several  DOD  sources.37  Further,  we 
have,  in  related  work,  applied  our  AOA  best  practices  to  AOA  processes 
that  concern  national  security  facilities.  In  these  audits,  we  have  generally 
found  that  agencies  did  not  fully  conform  to  AOA  best  practices.  In  most 
cases,  we  assessed  those  agencies’  AOA  processes  as  partially  meeting 
or  not  meeting  our  best  practices.  In  those  cases,  we  recommended  that 


36The  customer  refers  to  the  program  office,  service,  or  agency  that  identifies  a  mission 
need;  the  decision  maker  is  the  person  or  entity  that  signs  off  on  the  final  decision  and 
analysis  documented  by  the  AOA  report.  We  describe  the  role  of  the  customer  and 
decision  maker  in  appendix  III.  The  best  practices  that  include  the  customer  or  decision 
maker  are  Define  Mission  Need,  Define  Functional  Requirements,  Develop  AOA  Time 
Frame,  Define  Selection  Criteria,  Weight  Selection  Criteria,  and  Compare  Alternatives. 

37The  DOD  sources  include  the  following  guidance:  Air  Force,  Office  of  Aerospace 
Studies,  U.S.  Air  Force  Materiel  Command,  Analysis  of  Alternatives  (AOA)  Handbook:  A 
Practical  Guide  to  Analyses  of  Alternatives,  the  Defense  Acquisition  Guidebook,  DOD 
Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System  (Jan.  7,  2015),  and  the 
Army  Cost  Benefit  Guide,  3rd  Ed.  (Apr.  24,  2013).  In  appendix  II,  we  discuss  these 
sources  in  more  detail. 
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agencies  use  such  best  practices  in  the  future  and  agencies  have 
generally  concurred  with  our  recommendations.  For  example,  in  2014,  we 
conducted  an  audit  similar  to  our  analysis  of  DOD’s  AOA  process  for 
JIAC  consolidation,  examining  three  projects  in  which  the  National 
Nuclear  Security  Administration  designed  and  constructed  facilities.38  Our 
analysis  of  the  National  Nuclear  Security  Administration’s  facility  projects 
indicated  that  each  project’s  AOA  partially  met  the  best  practices  for 
conducting  an  AOA  process.  This  raised  concerns  about  the  reliability  of 
these  AOAs,  and  we  made  a  recommendation  that  the  agency  update  its 
project  management  requirements  by  incorporating  best  practices  for 
conducting  an  AOA.  The  National  Nuclear  Security  Administration 
concurred  and  has  begun  implementing  our  recommendation.  The  fact 
that  DOD’s  AOA  team  fully  or  substantially  met  six  of  our  best  practices 
indicates  that  subject  matter  experts  engaged  in  a  military  construction 
decision  making  process  can  follow  these  best  practices. 

Further,  the  principles  and  practices  contained  in  GAO’s  AOA  best 
practices  parallel  those  found  in  DOD  and  Air  Force  guidance  on  military 
construction  and  analysis  for  decision  making.  For  example,  according  to 
a  DOD  directive  governing  military  construction,  the  department  must 
monitor  the  execution  of  its  military  construction  program  to  ensure — 
among  other  things — the  most  cost-effective  accomplishment  of  the 
program.  Our  AOA  best  practice  “Develop  life-cycle  cost  estimates” 
focuses  on  providing  decision  makers  with  the  information  they  need  to 
assess  the  cost-effectiveness  of  alternatives.  Also,  according  to  an  Air 
Force  instruction  governing  the  planning  and  programing  for  military 
construction  projects,  one  of  the  required  planning  actions  is  to  evaluate 
alternative  solutions.39  This  guidance  on  planning  is  consistent  with  our 
AOA  best  practices,  in  which  we  state  that  the  best  practices  can  be 
applied  to  a  wide  range  of  activities  in  which  an  alternative  must  be 
selected  from  a  set  of  possible  options.  We  discuss  the  alignment  of  our 
best  practices  with  DOD  and  Air  Force  guidance  in  more  detail  in 
appendix  II. 


38GAO,  DOE  and  NNSA  Project  Management:  Analysis  of  Alternatives  Could  Be 
Improved  by  Incorporating  Best  Practices,  GAO-15-37  (Washington,  D.C.:  Dec.  11, 2014). 

39Air  Force  Instruction  32-1 021 ,  Planning  and  Programming  Military  Construction 
(MILCON)  Projects,  (Feb.  25,  2016). 
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Finally,  as  stated  earlier,  DOD  does  not  have  guidance  for  applying  AOA 
best  practices  when  making  military  construction  decisions.  According  to 
DOD  officials,  they  were  concerned  that  application  of  GAO’s  AOA  best 
practices  to  all  military  construction  projects  is  not  practical,  given  the 
number  of  personnel  and  financial  resources  that  the  officials  believe 
would  be  needed  to  conduct  the  analyses  required  by  the  best  practices. 
However,  DOD  could  apply  our  AOA  best  practices  for  certain  military 
construction  projects,  perhaps  establishing  a  threshold  for  the  use  of  the 
best  practices  in  military  construction  decisions  that  require  a  higher 
funding  level.  For  example,  DOD  could  apply  our  AOA  best  practices  for 
military  construction  projects  above  a  certain  funding  threshold — such  as 
the  JIAC — for  which  Air  Force  military  construction  guidance  encourages 
phasing  of  large  funded  projects  over  multiple  years.40  Without  guidance 
for  using  AOA  best  practices  when  conducting  AOA  assessments  for 
certain  military  construction  projects,  DOD  and  Congress  may  face 
oversight  challenges,  and  DOD  may  continue  to  have  trouble  justifying  its 
decisions  for  projects  in  the  future. 


Conclusions 


DOD’s  determination  that  it  needed  new  facilities  to  house  the  JIAC  was 
rooted  in  the  recognition  that  the  inadequacy  of  current  facilities  at  RAF 
Molesworth  had  operational  and  financial  impacts  affecting  the 
intelligence  center’s  ability  to  effectively  support  the  missions  of  EUCOM, 
AFRICOM,  and  U.S.  allies.  Consequently,  the  department  chose  criteria 
to  guide  its  AOA  team  in  selecting  a  preferred  course  of  action,  and  the 
reasoning  behind  DOD’s  eventual  decision  to  use  military  construction 
funding  to  build  new  facilities  at  RAF  Croughton  was  explained  in  DOD 
documentation  and  by  the  department’s  officials.  However,  our 
examination  of  their  AOA  process  against  GAO’s  best  practices  for 
conducting  an  analysis  of  alternatives  identified  shortcomings  in  DOD’s 
process.  As  officials  noted,  DOD  does  not  have  a  set  of  best  practices  for 
conducting  analysis  of  alternatives  for  military  construction  project 
decisions  and  the  AOA  team  did  not  follow  a  particular  piece  of  guidance 
or  previously  identified  best  practices  as  they  carried  out  their  analysis. 

As  a  result — especially  in  regard  to  documentation — DOD  was  not 


40See  Air  Force  Instruction  32-1021,  Planning  and  Programming  Military  Construction 
(MILCON)  Projects ,  which  states  that  OMB  Circular  A-11  encourages  phasing  of  large 
funded  projects  into  complete  and  usable  phases,  with  phases  in  multiple  years,  when 
feasible,  in  lieu  of  incrementing  projects. 
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optimally  positioned  to  explain  its  choice  of  RAF  Croughton.  In  our 
previous  work  assessing  AOA  processes  for  national  security  facilities, 
we  found  that  agencies  did  not  fully  conform  to  AOA  best  practices  and 
recommended  that  agencies  use  such  best  practices  in  the  future; 
agencies  generally  concurred  with  our  recommendations.  Without 
guidance  directing  that  military  construction  AOA  processes  be  conducted 
in  accordance  with  identified  best  practices,  DOD  will  not  be  positioned  to 
fully  provide  reasonable  assurance  that  decisions  in  similar  future  cases 
are  the  result  of  a  high-quality,  reliable  AOA  process  and  may  result  in 
difficulty  for  DOD  and  Congress  as  they  provide  oversight  over  these 
decisions. 


Recommendation  for 
Executive  Action 


To  aid  DOD  in  conducting  future  AOA  processes  that  fully  follow  best 
practices,  we  recommend  that  the  Secretary  of  Defense  direct  the 
Assistant  Secretary  of  Defense  for  Energy,  Installations,  and  the 
Environment  to  develop  guidance  requiring  the  use  of  AOA  best 
practices,  including  those  practices  we  have  identified,  and  in  this 
guidance,  the  Assistant  Secretary  should  define  the  types  of  military 
construction  decisions  for  which  these  AOA  best  practices  should  be 
required. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  the  classified  version  of  this  report  to  DOD  for 
review  and  comment;  the  department  provided  technical  comments  that 
we  considered  and  incorporated  as  appropriate.  DOD  also  provided 
written  comments  on  our  recommendation,  which  are  reprinted  in 
appendix  V. 

In  its  written  comments,  DOD  did  not  concur  with  our  recommendation. 
Specifically,  DOD  disputes  that  our  22  best  practices  for  a  reliable  AOA 
process  apply  to  basing  or  military  construction  decision-making 
processes  and  therefore  does  not  believe  that  the  department  should 
incorporate  these  best  practices  into  its  military  construction  decision¬ 
making  process. 

We  continue  to  believe  that  our  AOA  best  practices  can  be  applied  to  a 
wide  range  of  activities  in  which  an  alternative  must  be  selected  from  a 
set  of  possible  options,  as  well  as  to  a  broad  range  of  capability  areas, 
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projects,  and  programs — including  DOD’s  military  construction  decision¬ 
making  processes.  As  discussed  in  our  report,  during  GAO’s 
development  of  our  AOA  best  practices,  we  reviewed  a  variety  of  U.S. 
government  guidance,  including  several  DOD  sources.41  Also,  during 
discussions  with  DOD  officials  over  the  course  of  our  audit,  officials  stated 
that  they  see  the  value  of  our  AOA  best  practices.  In  addition,  when  DOD 
provided  comments  on  our  draft  analysis  of  its  AOA  process,  department 
officials  demonstrated  through  their  comments  that  they  believe  the  JIAC 
AOA  team  was  able  to  follow  almost  all  of  the  best  practices  and  meet  the 
practices’  requirements.42 

Further,  in  its  written  comments,  DOD  states  that  it  does  not  believe  our 
22  AOA  best  practices  are  applicable  to  basing  decisions  in  an 
operational  theater  that  must  take  into  account  a  number  of  subjective 
factors  such  as  best  military  judgment.  However,  as  discussed  in  our 
report,  our  AOA  best  practices  incorporate  the  professional  judgment  of 
senior  agency  officials.  Our  assessment  that  the  JIAC  AOA  team  fully  met 
two  best  practices  that  specifically  include  officials’  professional  judgment 
is  evidence  that  DOD  can  follow  these  AOA  best  practices  while  its 
leaders  exercise  their  professional  military  judgment.43  Thus,  we  continue 
to  believe  that  our  AOA  best  practices  should  be  applied  to  a  variety  of 
decision-making  processes,  including  those  that  involve  professional 
judgment. 


41 

These  sources  include  the  Air  Force  AOA  Handbook  discussed  previously;  the  Defense 
Acquisition  Guidebook  (Defense  Acquisition  University,  The  Defense  Acquisition 
Guidebook ,  accessed  January  20,  2016,  https://dag.dau.mil/Pages/Default.aspx.); 
Department  of  Defense,  Operation  of  the  Defense  Acquisition  System ,  DOD  Instruction 
5000.02,  (Jan.  7,  2015);  and  Army,  Army  Cost  Benefit  Analysis  Guide  (Apr.  24,  2013).  We 
discuss  these  sources  further  in  appendix  II. 

42 As  discussed  in  appendix  II,  we  incorporated  these  comments  into  our  final  scoring  of 
the  JIAC  AOA  process. 

43As  discussed  in  our  report,  according  to  the  best  practices,  two  of  the  key  entities 
involved  in  the  AOA  process — the  customer  and  decision  maker — are  expected  to 
exercise  their  professional  judgment  in  making  key  decisions.  For  example,  in  our  best 
practice  of  Defining  Mission  Need,  the  customer  defines  the  gap  between  current 
capabilities  and  those  required  to  meet  the  agency’s  goals.  Also,  for  our  best  practice 
Define  Selection  Criteria,  the  AOA  team  or  the  decision  maker  defines  selection  criteria 
based  on  the  mission  need. 
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In  addition,  in  its  written  comments,  DOD  explained  that  it  took  issue  with 
our  use  of  the  AOA  best  practices  because  the  practices  were  published 
after  the  department  finished  the  JIAC  AOA  process.  While  this  is  the 
case,  our  AOA  best  practices  are  based  on  longstanding,  fundamental 
tenants  of  sound  decision  making  and  economic  analysis.  The  best 
practices  include  steps  such  as  an  agency  defining  its  mission  needs, 
developing  a  list  of  alternatives,  assessing  those  alternatives’  viability, 
and  ensuring  that  its  AOA  process  is  impartial.  When  we  asked  DOD  to 
provide  a  list  of  specific  AOA  best  practices  that  the  department  believes 
do  not  apply  to  the  basing  or  military  construction  decision-making 
processes,  the  department  declined  to  identify  the  AOA  best  practices 
that  it  believes  do  not  apply.  Further,  as  discussed  in  this  report,  the 
principles  and  practices  contained  in  our  AOA  best  practices  parallel 
those  found  in  DOD  and  Air  Force  guidance  on  military  construction  and 
economic  analysis  for  decision  making. 

In  its  written  comments,  DOD  also  explained  that  it  does  not  concur  with 
our  recommendation  that  the  department  develop  guidance  requiring  the 
use  of  AOA  best  practices,  including  those  practices  we  have  identified. 
However,  we  continue  to  believe  that  our  recommendation  would  assist 
DOD  in  fully  providing  reasonable  assurance  that  decisions  in  similar 
future  cases  are  the  result  of  a  high-quality,  reliable  AOA  process.  This  is 
because,  in  our  assessment  of  DOD’s  AOA  process  for  JIAC 
consolidation,  we  found  that  the  department’s  process  only  partially  met 
each  of  the  four  characteristics  of  a  high-quality,  reliable  AOA  process.  In 
our  previous  work  on  AOA  processes,  we  have  found  that  not  fully 
conforming  to  the  best  practices  may  lead  to  an  unreliable  AOA,  and  in 
those  cases,  the  agency  will  not  be  positioned  to  ensure  that  the 
preferred  solution  best  meets  the  mission  needs. 

Explaining  its  reasons  for  not  concurring  with  our  recommendation,  DOD 
stated  that  the  JIAC  consolidation  military  construction  project  was 
developed  and  communicated  to  the  Congress  in  a  way  that  is  consistent 
with  both  the  statute  governing  DOD’s  military  construction  projects  and 
the  department’s  “long-standing,  supporting  policies.”44  For  this  report,  we 
were  asked  to  review  the  key  considerations  that  influenced  DOD’s 
decision  to  consolidate  the  JIAC  at  RAF  Croughton  and  the  extent  to 


44Section  2802(c)  of  title  10,  United  States  Code. 
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which  DOD’s  AOA  process  for  JIAC  consolidation  aligns  with  best 
practices  for  such  analyses.  Thus,  we  did  not  review  DOD’s  compliance 
with  the  statute.  As  discussed  in  our  report,  the  existing  policies  that  DOD 
followed  in  making  its  decision  on  the  location  for  JIAC  consolidation  were 
not  sufficient  to  ensure  a  fully  reliable  AOA  process.  For  example,  DOD 
officials  explained  that  they  do  not  believe  they  needed  to  fully  adhere  to 
our  AOA  best  practices  during  their  AOA  process  because  the  decision  to 
fund  the  JIAC  consolidation  military  construction  project  was 
considered — and  then  approved — through  the  department’s  Program, 
Budget,  and  Review  process.  As  we  note  in  our  report,  we  assessed 
documentation  from  that  process  for  our  review,  but  found  that  DOD’s 
Program,  Budget,  and  Review  process  was  not  a  substitute  for 
conducting  a  robust  AOA  for  JIAC  consolidation.  This  is  because — 
according  to  DOD  officials — DOD’s  Program,  Budget,  and  Review 
process  for  the  JIAC  consolidation  project  occurred  at  the  end  of  DOD’s 
AOA  process.  Further,  during  the  Program,  Budget,  and  Review  process, 
the  department  considered  only  the  preferred  solution  of  RAF  Croughton. 
Thus,  the  Program,  Budget  and  Review  process  cannot  serve  as  a 
substitute  for  an  AOA  process  that  adheres  to  all  of  the  AOA  best 
practices.  In  addition,  according  to  officials  from  the  Office  of  the 
Secretary  of  Defense  and  Air  Force  Headquarters,  DOD  does  not  have  a 
set  of  best  practices  for  conducting  an  AOA  for  military  construction 
project  decisions. 

Because  DOD’s  existing  processes  may  not  be  sufficient  to  ensure  fully 
reliable  AOA  processes  in  the  future — and  DOD  officials  acknowledged 
that  the  department  does  not  have  AOA  best  practices  for  basing  or 
military  construction  decisions — we  continue  to  believe  that  implementing 
our  recommendation  would  assist  DOD  in  ensuring  that,  in  the  future,  the 
department  makes  robust  decisions  concerning  key  facilities.  Also, 
because  our  recommendation  is  that  the  Assistant  Secretary  of  Defense 
for  Energy,  Installations,  and  the  Environment  should — in  guidance — 
define  the  types  of  military  construction  decisions  for  which  AOA  best 
practices  should  be  required,  we  believe  that  our  recommendation 
provides  DOD  with  reasonable  flexibility  to  incorporate  the  AOA  best 
practices  into  the  department’s  existing  processes. 

Without  guidance  directing  that  certain  military  construction  AOA 
processes  be  conducted  in  accordance  with  identified  best  practices,  the 
Congress  may  not  have  complete  information  to  inform  its  oversight  of 
DOD’s  future  military  construction  decisions.  This  conclusion  is  based  on 
our  assessment  of  DOD’s  AOA  process  for  JIAC  consolidation. 
Specifically,  that  DOD’s  lack  of  full  AOA  documentation  may  make  it 
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difficult  for  decision  makers  in  both  the  department  and  Congress  to  fully 
assess  how  DOD  arrived  at  its  final  decision  to  consolidate  the  JIAC  at 
RAF  Croughton,  as  oversight  on  this  issue  continues.  For  example,  in 
May  2016,  the  House  of  Representatives  passed  H.R.  4909. 45  Section 
1623  of  the  bill  would,  if  enacted,  limit  DOD’s  fiscal  year  2017  obligation 
or  expenditure  of  funding  for  intelligence  manpower  positions  for  JIAC 
operation  until  the  Secretary  of  Defense  provides  the  Congress  with — 
among  other  things — a  revised  AOA  for  JIAC  basing  that  is  informed  by 
the  findings  of  our  report  and  uses  best  practices.  Because  our  AOA  best 
practices  call  for  robust  documentation  as  part  of  the  AOA  process, 
implementing  our  recommendation  would  assist  DOD  in  providing  the 
Congress  with  the  type  of  information  that  could  better  meet  the 
Congress’  future  oversight  needs,  for  similar  types  of  projects. 


As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
date  of  this  report.  At  that  time,  we  will  send  copies  of  this  report  to  the 
appropriate  congressional  committees  and  to  the  Secretaries  of  Defense, 
the  Army,  the  Navy,  and  the  Air  Force;  the  Commandant  of  the  Marine 
Corps;  and  the  Assistant  Secretary  of  Defense  for  Energy,  Installations, 
and  Environment.  In  addition,  this  report  will  be  available  at  no  charge  on 
the  GAO  website  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
me  at  (202)  512-4523  or  leporeb@gao.gov.  Contact  points  for  our  Offices 


45H.R.  4909  is  the  House  of  Representatives’  version  of  the  bill  for  the  National  Defense 
Authorization  Act  for  fiscal  year  201 7.  The  Senate  bill  does  not  have  a  similar  provision 
and  if  enacted,  would  authorize  $53.1  million  in  funding  for  Phase  3  of  JIAC  consolidation 
at  RAF  Croughton.  See  S.  2943,  §  4601  passed  by  the  Senate  on  June  14,  2016. 
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of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  who  made  key  contributions  to  this  report 
are  listed  in  appendix  VI. 


Brian  J.  Lepore 
Director 

Defense  Capabilities  and  Management 


Page  45 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


List  of  Requesters 


The  Honorable  John  S.  McCain 
Chairman 

Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Mark  Kirk 
Chairman 

The  Honorable  Jon  Tester 
Ranking  Member 

Subcommittee  on  Military  Construction,  Veterans  Affairs, 
and  Related  Agencies 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Jason  Chaffetz 
Chairman 

Committee  on  Oversight  and  Government  Reform 
House  of  Representatives 

The  Honorable  Devin  Nunes 
Chairman 

Permanent  Select  Committee  on  Intelligence 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 


Page  46 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


Appendix  I:  Timeline  of  Key  Joint  Intelligence 
Analysis  Complex  (JIAC)  Consolidation 
Events,  2009-2016 


Figure  2:  Key  Events  for  JIAC  Consolidation  at  Royal  Air  Force  (RAF)  Base  Croughton,  2009  -  2016 


DOD’s  Analysis  and  Decisions  on  Joint  Intelligence  Analysis  Complex  (JIAC)  Consolidation 


September:  EUCOM  has 
initial  discussions  with  the 
Office  of  the  Secretary  of 
Defense  (OSD)  on  its 
concept  for  recapitalization 
of  the  JIAC.  OSD  instructs 
EUCOM  to  expand  scope 
of  locations  being 
considered.  Lajes  Field, 
Azores  was  not 
considered. 


September:  Issue  paper 
proposing  JIAC 
consolidation  project  is 
submitted;  not  approved. 
Review  within  DOD 
continues. 


February:  In  the  submission  of 
DOD’s  budget  for  Fiscal  Year 
2013,  Air  Force  includes  funding 
for  planning  and  design  of  JIAC 
consolidation  at  Royal  Air  Force 
(RAF)  base  Croughton. 


October:  Revised  issue 
paper  is  submitted; 
approved  by  3-star  and 
4-star  review  panels. 


April:  Resource 
Management  Decision 
issued  by  DOD. 


European  Infrastructure  Consolidation  (EIC)  study 


2013  f  2014  ,  2015  ,  2016  ,  2017 


T  T 

January:  DOD  begins  EIC.  January:  DOD  completes  EIC, 

incorporating  its  previous  decision 
to  close  two  RAF  bases,  enabled 
by  JIAC  consolidation  at  RAF 
Croughton.  DOD  also  decides  to 
streamline  Lajes  Field  operations 
due  to  cost  savings  enabled  by 
decreased  mission  requirements. 


JIAC  Consolidation  Authorization  and  Appropriations 


2014  (  2015  (  2016  |  2017 


l _ T 


December:  Congress  authorizes  and 
appropriates  $92.2  million  for  Phase  1  of 
JIAC  consolidation.  Congress  requires  that 
DOD— before  obligating  or  expending  these 
funds — complete  the  EIC;  provide  Congress 
with  information  on  DOD’s  European  basing 
strategy;  and  certify  that  the  requirement  for 
any  military  construction  authorized  in  the 
U.K.  is  identified  in  the  EIC. 


November:  Congress  authorizes  $94.2  million  for 
Phase  2  of  JIAC  consolidation.  But,  Congress  requires 
DOD  to  certify  that  RAF  Croughton  remains  the  optimal 
location  for  JIAC  consolidation  before  DOD  expends 
funds  on  construction  of  Phase  2,  and  also,  provide 
Congress  with  additional  information  on  Lajes  Field.  In 
addition,  Congress  appropriated  funding  for  Phase  2  of 
the  JIAC  but  restricted  DOD’s  use  of  other  funding — to 
realign  forces  at  Lajes  Field — until  DOD  certified  that 
Lajes  Field  is  not  an  optimal  location  for  the  JIAC. 


JIAC  Consolidation  Project  Execution 


2015  ,  2016  ,  2017 


August:  DOD 

obligates  $92.2  million 
to  United  Kingdom  for 
Phase  1  of  JIAC 
consolidation. 


July:  DOD  obligates 
$94.2  million  to 
United  Kingdom  for 
Phase  2  of  JIAC 
consolidation. 


T 

Fall: 

Construction  is 
planned  to  begin 
on  Phase  1  and 
Phase  2. 


Source:  GAO  analysis  of  Department  of  Defense  (DOD)  information  and  statements  from  officials.  I  GAO-16-853 
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Appendix  II:  Scope  and  Methodology 


In  order  to  address  our  research  objectives,  we  assessed  information 
from  various  Department  of  Defense  (DOD)  organizations,  including  U.S. 
Air  Force  Headquarters,  the  Basing  Directorate  of  the  Office  of  the 
Secretary  of  Defense,  and  the  headquarters  of  both  the  U.S.  European 
and  Africa  Commands  (EUCOM  and  AFRICOM).1  For  the  purposes  of 
this  audit,  these  organizations  made  up  DOD’s  “AOA  team”  for  the  AOA 
process  for  consolidation  of  its  Joint  Intelligence  Analysis  Complex 
(JIAC).  This  a  term  that  we  use  to  describe  the  key  organizations  that 
contained  the  subject  matter  experts  who  were  involved  in  the  day-to-day 
work  of  the  AOA  process  and  worked  to  develop  the  analysis  that  was  the 
foundation  of  the  AOA  process.2  We  also  visited  various  components  of 
the  JIAC  at  Royal  Air  Force  (RAF)  base  Molesworth,  the  JIAC’s  support 
facilities  located  at  RAF  Alconbury,  and  the  planned  future  JIAC  site  at 
RAF  Croughton. 


Considerations  That 
Influenced  DOD’s  Decision 
to  Consolidate  the  JIAC  at 
RAF  Croughton 


To  describe  the  considerations  that  influenced  DOD’s  decision  to 
consolidate  the  JIAC  at  RAF  Croughton,  we  reviewed  various  DOD 
documents  describing  the  condition  of  the  JIAC’s  current  facilities, 
including  Air  Force  budget  justification  documents  submitted  to  the 
Congress  for  the  JIAC  consolidation  military  construction  projects.  During 
our  site  visit  to  the  various  components  of  the  JIAC  at  RAF  Molesworth 
and  its  support  facilities  located  at  RAF  Alconbury,  we  met  with  JIAC 
officials  to  learn  about  the  facilities’  condition,  discussing — among  other 
things — how  their  condition  can  affect  financial  and  operational  impacts. 
In  order  to  describe  the  five  criteria  DOD  applied  in  its  decision  to 
consolidate  the  JIAC  at  RAF  Croughton — and  key  issues  related  to  those 


1 0ther  organizations  with  which  we  met,  or  from  which  we  gathered  information,  include 
the  Office  of  the  Director  National  Intelligence,  the  Defense  Intelligence  Agency,  and 
Defense  Information  Systems  Agency. 

2Over  the  course  of  our  investigation,  we  asked  for  original  data  and  documentation  that 
the  AOA  used  in  its  analyses.  In  lieu  of  available  data  or  documentation,  we  asked  to 
speak  with  DOD  officials  who  participated  in  the  AOA  process.  DOD  was  able  to  provide 
several  officials  who  participated  in  the  process.  However,  this  was  not  always  the  case. 
According  to  DOD  officials  with  whom  we  spoke,  certain  personnel  who  may  have 
participated  in  the  AOA  process  were  not  available.  The  officials  provided  various 
reasons,  such  as  contractors  who  no  longer  worked  for  the  relevant  DOD  organization, 
civilians  who  had  retired,  or  military  personnel  who  rotated  to  different  assignments.  DOD 
officials  explained  that,  given  the  AOA  process  began  in  2009  and  military  personnel 
typically  rotate  on  three-year  assignments,  these  personnel  could  be  on  their  second  or 
third  rotation  since  working  as  part  of  the  AOA  team. 
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criteria — we  reviewed  DOD  documentation  from  its  AOA  process,  such  as 
a  201 1  preliminary  summary  of  the  process;  interviewed  officials  from 
DOD’s  AOA  team;  and  reviewed  additional,  related  documents,  such  as 
Air  Force  guidance  that  address  requirements  for  the  use  of  temporary 
facilities.* * 3 


The  Extent  to  Which 
DOD’s  Analysis  of 
Alternatives  Process  for  Its 
JIAC  Consolidation  Project 
Aligns  with  Best  Practices 
for  Such  Analyses 


To  determine  the  extent  to  which  DOD’s  analysis  of  alternatives  process 
for  its  JIAC  consolidation  project  aligns  with  best  practices  for  such 

analyses,  we  reviewed  all  data  and  documentation  developed  by  DOD  as 
a  part  of  its  AOA  process  from  the  initial  concept  proposal  of  JIAC 
consolidation  in  the  fall  of  2009  to  the  Resource  Management  Decision 
for  JIAC  consolidation,  issued  by  the  Secretary  of  Defense  in  April  2013. 
We  refer  to  this  collection  of  information  (produced  from  the  fall  of  2009  to 
the  spring  of  2013)  as  DOD’s  “AOA  body  of  work.”  In  this  study,  we  cite 
key  documents  from  DOD’s  body  of  work  with  abbreviated  titles.  For 
example,  in  the  fall  of  201 1 ,  EUCOM  produced  a  preliminary  summary  of 
DOD’s  AOA  efforts — up  to  that  point  in  the  process — named  Analysis  of 
Alternatives  (AOA)  for  Recapitalization  of  Intelligence  Facilities  at  RAF 
Molesworth.  We  refer  to  this  document  as  the  “201 1  preliminary  AOA 
summary.”  In  addition,  we  discussed  DOD’s  AOA  process  for  JIAC 
consolidation  with  officials  from  organizations  in  the  AOA  team  and  at  the 
current  JIAC.  After  collecting  available  data  and  documentation  from 
DOD,  we  evaluated  DOD’s  AOA  body  of  work  against  GAO’s  22  AOA 
best  practices.  In  appendix  III,  these  best  practices  are  described  in 
detail.  Their  applicability  to  DOD’s  military  construction  process  is 
discussed  in  more  detail  below.4  We  then  scored  DOD’s  AOA  body  of 


3Air  Force  Instruction  32-1032,  Planning  and  Programming  Appropriated  Fund 
Maintenance,  Repair,  and  Construction  Projects  (Sept.  24,  2015). 

4These  best  practices  were  originally  published  in  GAO,  Amphibious  Combat  Vehicle: 
Some  Acquisition  Activities  Demonstrate  Best  Practices;  Attainment  of  Amphibious 
Capability  to  Be  Determined,  GAO-16-22,  (Washington,  D.C.:  Oct.  28,  2015).  To  evaluate 
DOD’s  AOA  process,  we  took  the  following  steps:  (1)  two  GAO  analysts  separately 
examined  the  AOA  information  received  from  DOD,  providing  a  score  for  each  of  the  22 
best  practices;  (2)  a  third  GAO  analyst  adjudicated  any  differences  between  the  two 
analysts’  initial  scoring;  (3)  a  GAO  specialist  on  AOA  best  practices,  independent  of  the 
audit  team,  reviewed  the  team’s  adjudicated  AOA  documentation  and  scores,  cross¬ 
checking  the  scores  and  all  of  the  analyses  for  consistency. 
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work  against  each  best  practice.5  The  team  used  the  average  of  the 
scores  for  each  of  the  individual  best  practices  to  determine  an  overall 
score  for  the  four  summary  characteristics  for  a  reliable  AOA  process — 
well-documented,  comprehensive,  unbiased,  and  credible.6  Next  we 
shared  our  draft  analysis  with  DOD,  asking  that  the  department  provide 
technical  comments  and  any  additional  documentation  that  might  impact 
our  assessment.  We  then  incorporated  these  additional  comments  and 
additional  documentation  to  ensure  our  analysis  included  all  available 
information.  Finally,  we  applied  the  same  methodology  and  scoring 
process  explained  above  to  revise  our  initial  analysis  based  on  DOD’s 
technical  comments  and  any  additional  evidence  received.  For  those 
characteristics  of  the  AOA  process  that  received  a  lower  than  average 
score,  we  met  with  DOD  officials  to  discuss  potential  reasons  why  they 
did  not  conform  to  best  practices  for  those  parts  of  the  AOA  process. 
Examining  DOD’s  AOA  process  for  JIAC  consolidation  with  our  AOA  best 
practices  allowed  us  to  assess  the  strengths  and  weaknesses  of  the 
department’s  process.  Our  best  practices  were  not  used  to  determine 
whether  DOD  made  the  correct  decision  on  the  location  for  JIAC 
consolidation  or  whether  the  department  would  have  arrived  at  a  different 
conclusion  had  it  more  fully  conformed  to  our  best  practices.  Rather,  we 
used  our  best  practices  to  assess  the  degree  to  which  DOD  can  provide 
reasonable  assurance  that  its  process  met  each  of  the  four  characteristics 
of  a  high-quality,  reliable  AOA  process.  In  the  course  of  applying  our  AOA 
best  practices  to  information  in  DOD’s  AOA  body  of  work,  we  assessed 
the  reasonableness  of  the  information  we  collected.  We  determined  that 
the  information  from  DOD’s  AOA  body  of  work  is  sufficiently  reliable  for 
the  purposes  of  describing  DOD’s  rationale  for  choosing  RAF  Croughton 


5GAO’s  best  practices  define  five  different  qualitative  and  quantitative  categories  for 
scoring.  The  five-point  qualitative  system  we  used  is  as  follows.  Fully  Meets:  DOD 
provided  complete  evidence  that  satisfies  the  elements  of  the  best  practice;  Substantially 
Meets:  DOD  provided  evidence  that  satisfies  a  large  portion  of  the  elements  of  the  best 
practice;  Partially  Meets:  DOD  provided  evidence  that  satisfies  about  half  of  the  elements 
of  the  best  practice;  Minimally  Meets:  DOD  provided  evidence  that  satisfies  a  small  portion 
of  the  elements  of  the  best  practice;  and  Does  Not  Meet:  DOD  provided  no  evidence  that 
satisfies  any  of  the  elements  of  the  best  practice.  The  corresponding  quantitative 
categories  are  as  follows.  Not  Met  =  1,  Minimally  Met  =  2,  Partially  Met  =  3,  Substantially 
Met  =  4,  and  Fully  Met  =  5. 

6The  resulting  average  score,  for  each  characteristic,  corresponds  to  one  of  the  five 
qualitative  categories,  as  follows:  Not  Met  =  1.0  to  1.4,  Minimally  Met  =  1.5  to  2.4,  Partially 
Met  =  2.5  to  3.4,  Substantially  Met  =  3.5  to  4.4,  and  Fully  Met  =  4.5  to  5.0. 
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as  the  location  for  JIAC  consolidation  and  comparing  DOD’s  AOA 
process  to  our  22  best  practices  for  a  reliable  AOA  process. 


Applicability  of  GAO’s  AOA 
Best  Practices  to  Decision 
Making  in  DOD’s  Military 
Construction  Process 


In  GAO’s  development  of  our  AOA  best  practices,  we  reviewed  a  variety 
of  U.S.  government  guidance,  including  several  DOD  sources.7  These 
DOD  sources  include  the  Air  Force  AOA  Handbook  discussed 
previously,8  the  Defense  Acquisition  Guidebook,9  DOD’s  Instruction  on 
Operation  of  the  Defense  Acquisition  System  (DODI  5000.02), 10  and  the 
Army  Cost  Benefit  Analysis  Gu/'de.11  We  have  previously  reported  that  our 
AOA  best  practices  can  be  applied  to  a  wide  range  of  activities  in  which 
an  alternative  must  be  selected  from  a  set  of  possible  options,  as  well  as 
to  a  broad  range  of  capability  areas,  projects,  and  programs.12  Further, 
the  principles  and  practices  contained  in  GAO’s  AOA  best  practices 
parallel  those  found  in  DOD  and  Air  Force  guidance  on  military 
construction  and  economic  analysis  for  decision  making.  The  congruence 
between  our  AOA  best  practices  and  the  department’s  guidance  on  these 
subjects  demonstrates  the  best  practices’  applicability  to  AOA  processes 
for  military  construction  projects  such  as  JIAC  consolidation. 


For  example,  according  to  a  DOD  directive  governing  military 
construction,  the  Assistant  Secretary  of  Defense  for  Energy,  Installations 
and  Environment  must  monitor  the  execution  of  its  military  construction 
program  to  ensure — among  other  things — the  most  cost-effective 
accomplishment  of  the  program.13  This  principal  of  cost  effectiveness 


7For  an  explanation  of  the  AOA  process  and  description  of  each  best  practice,  see 
appendix  III. 

8Air  Force,  Office  of  Aerospace  Studies,  U.S.  Air  Force  Materiel  Command,  Analysis  of 
Alternatives  (AOA)  Handbook:  A  Practical  Guide  to  Analyses  of  Alternatives  (June  2013). 

9Defense  Acquisition  University,  The  Defense  Acquisition  Guidebook,  accessed  January 
20,  2016,  https://dag.dau.mil/Pages/Default.aspx. 

10DOD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System  (Jan.  7,  2015). 
11  Army,  Army  Cost  Benefit  Analysis  Guide  (Apr.  24,  2013). 

12GAO,  Amphibious  Combat  Vehicle:  Some  Acquisition  Activities  Demonstrate  Best 
Practices;  Attainment  of  Amphibious  Capability  to  be  Determined,  GAO- 16-22 
(Washington,  D.C.:  Oct.  28,  2015). 

13DOD  Directive  4270.5,  Military  Construction  (Feb.  12,  2005). 
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aligns  with  GAO’s  AOA  best  practice  Develop  Life-cycle  Cost  Estimates, 
which  focuses  on  providing  decision  makers  with  the  information  they 
need  to  assess  the  cost-effectiveness  of  alternatives.  Further,  DOD 
Instruction  7041 .03,  on  economic  analysis  for  decision  making,  contains 
numerous  procedures  that  closely  parallel  those  called  for  in  our  AOA 
best  practices.14  There  are  several  examples  that  demonstrate  how  this 
instruction  parallels  our  AOA  best  practices. 

•  According  to  DOD  Instruction  7041 .03,  the  statement  of  the  objective 
should  not  assume  a  specific  means  of  achieving  the  desired  result.  If 
such  an  assumption  is  made,  the  statement  of  the  objective 
undermines  the  analytical  purpose  of  the  economic  analysis  by 
prejudging  the  result  and  should  be  avoided.  Our  AOA  best  practice 
Define  Mission  Need  states  that  the  agency  defines  the  mission 
needs  (i.e.,  a  credible  gap  between  current  capabilities  and  those 
required  to  meet  the  goals  articulated  in  the  strategic  plan)  without  a 
predetermined  solution  and  that  allowing  mission  needs  to  be  defined 
in  solution-specific  terms  creates  a  potential  bias  and  could  invalidate 
the  analysis. 

•  DOD  Instruction  7041 .03  states  that  all  reasonable  ways  of  satisfying 
the  objective  must  be  documented  and  discussed;  further,  careful 
attention  must  be  given  to  identifying  alternatives.  According  to  our 
AOA  best  practice  Develop  List  of  Alternatives,  the  AOA  team  should 
identify  and  consider  a  diverse  range  of  alternatives  to  meet  the 
mission  need;  and  that  an  AOA  process  encompasses  numerous 
alternatives  in  order  to  ensure  that  the  study  provides  a  broad  view  of 
the  issue. 

•  DOD  Instruction  7041.03  also  states  that  an  economic  analysis  should 
include  a  recommendation  of  the  preferred  alternative;  further,  the 
results  of  the  economic  analysis — including  all  calculations  and 
sources  of  data — must  be  documented  down  to  the  most  basic  inputs 
to  provide  an  auditable  and  stand-alone  document.  According  to  our 
AOA  best  practice  Document  AOA  Process  in  a  Single  Document,  the 
AOA  team  should  produce  a  document  that  clearly  states  the 
preferred  alternative,  and  documents  all  steps  taken  to  identify, 
analyze,  and  select  alternatives. 


1 4 

DOD  Instruction  7041.03,  Economic  Analysis  for  Decision-making  (Sept.  9,  2015). 
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There  are  also  several  examples  that  demonstrate  how  our  AOA  best 
practices  parallel  Air  Force  guidance.  For  instance,  Air  Force  Policy 
Directive  32-10,  which  establishes  policy  for  Air  Force  Installations  and 
Facilities,  states  that  its  civil  engineers  should — among  other  things — 
reduce  the  life-cycle  costs  of  facilities  by  making  the  full  costs  of  products 
and  services  visible  to  customers.15  This  principle  corresponds  with  the 
intent  of  our  AOA  best  practice  Develop  Life-cycle  Cost  Estimates, 
which — as  discussed  above — requires  the  AOA  team  to  develop  a  life- 
cycle  cost  estimate  for  each  alternative,  including  all  of  the  project’s  costs. 
Also,  according  to  Air  Force  Instruction  32-1021  on  the  planning  and 
programing  for  military  construction  projects,  one  of  the  required  planning 
actions  is  to  evaluate  alternative  solutions.16  This  guidance  on  planning 
agrees  with  the  intent  of  our  AOA  best  practices,  in  which  we  state  that 
the  best  practices  can  be  applied  to  a  wide  range  of  activities  in  which  an 
alternative  must  be  selected  from  a  set  of  possible  options.  Further,  Air 
Force  Instruction  32-1021  states  that  existing  assets  are  to  be  evaluated 
in  order  to  determine  the  most  economical  and  effective  means  of 
satisfying  facility  needs.  Our  AOA  best  practices  parallel  this  guidance. 
Specifically,  our  AOA  best  practice  Include  Baseline  Alternative  requires 
the  AOA  team  to  include  one  alternative  that  represents  the  status  quo  to 
provide  a  basis  of  comparison  among  alternatives. 

In  addition,  Air  Force  Instruction  65-501  on  economic  analysis  states  that 
economic  analysis  must  be  done  for  military  construction  projects  before 
an  alternative  has  been  chosen,  i.e.  during  the  AOA  process.17  Several  of 
our  AOA  best  practices  closely  align  with  the  specific  requirements  of  this 
economic  analysis  for  Air  Force  military  construction  projects.  For 
example,  according  to  this  Air  Force  instruction,  the  project  cost  estimate 
for  every  alternative  in  an  economic  analysis  must  be  of  the  same  quality 
and  accuracy,  and  will  normally  use  the  same  estimating  tool  or  method 
as  was  used  for  the  originally  proposed  project.  Our  AOA  best  practice 
Develop  AOA  Process  Plan  echoes  this  guidance,  requiring  that  the  AOA 
team  create  a  plan  to  include — among  other  things — the  proposed 


15Air  Force  Policy  Directive  32-10,  Civil  Engineering:  Installations  and  Facilities  (Mar.  4, 
2010). 

16Air  Force  Instruction  32-1 021 ,  Civil  Engineering:  Planning  and  Programming  Military 
Construction  Projects  (Feb.  25,  2016). 

17Air  Force  Instruction  65-501 ,  Economic  Analysis  (Aug.  29,  201 1 ). 
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methodologies  for  analyzing  alternatives  and  measures  that  are  used  to 
rate,  rank,  and  decide  among  the  alternatives.  Air  Force  Instruction  65- 
501  also  states  that  fair,  unbiased  and  accurate  cost  estimates  for  each 
alternative  in  an  economic  analysis  are  essential  to  a  fair  and  reasonable 
comparison  of  alternatives.  Our  AOA  best  practice  Perform  Independent 
Review  parallels  the  instruction.  Specifically,  our  best  practices  state  that 
the  AOA  process  should  be  an  unbiased  inquiry  into  the  costs,  benefits, 
and  capabilities  of  all  alternatives  and  that  an  independent  review  is  one 
of  the  most  reliable  means  to  validate  an  AOA  process.  Without  such  a 
review,  the  AOA  results  are  more  likely  to  include  organizational  bias  or 
lack  the  thoroughness  needed  to  ensure  that  a  preferred  solution — as 
opposed  to  a  favored  solution — is  chosen. 


Page  54 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


Appendix  III:  Best  Practices  for  the  Analysis 
of  Alternatives  Process 


Background  and 
Introduction 


Many  guides  have  described  an  approach  to  Analysis  of  Alternatives 
(AOA);  however,  there  is  no  single  set  of  practices  for  the  AOA  process 
that  has  been  broadly  recognized  by  both  the  government  and  private- 
sector  entities.  GAO  has  identified  22  best  practices  for  an  AOA  process 
by  (1)  compiling  and  reviewing  commonly  mentioned  AOA  policies  and 
guidance  used  by  different  government  and  private-sector  entities  and  (2) 
incorporating  experts’  comments  on  a  draft  set  of  practices  to  develop  a 
final  set  of  practices.1 

These  practices  can  be  applied  to  a  wide  range  of  activities  in  which  an 
alternative  must  be  selected  from  a  set  of  possible  options,  as  well  as  to  a 
broad  range  of  capability  areas,  projects,  and  programs.  These  practices 
can  provide  a  framework  to  help  ensure  that  entities  consistently  and 
reliably  select  the  project  alternative  that  best  meets  mission  needs.  The 
guidance  below  is  meant  as  an  overview  of  the  key  principles  that  lead  to 
a  successful  AOA  process  and  not  as  a  “how  to”  guide  with  detailed 
instructions  for  each  best  practice  identified. 

The  22  best  practices  that  GAO  identified  are  grouped  into  the  following 
five  phases: 

1 .  Initialize  the  AOA  process:  includes  best  practices  that  are  applied 
before  starting  the  process  of  identifying,  analyzing,  and  selecting 
alternatives.  This  includes  determining  the  mission  need  and 
functional  requirements,  developing  the  study  time  frame,  creating  a 
study  plan,  and  determining  who  conducts  the  analysis. 

2.  Identify  alternatives:  includes  best  practices  that  help  ensure  the 
alternatives  to  be  analyzed  are  sufficient,  diverse,  and  viable. 

3.  Analyze  alternatives:  includes  best  practices  that  compare  the 
alternatives  to  be  analyzed.  The  best  practices  in  this  category  help 
ensure  that  the  team  conducting  the  analysis  uses  a  standard, 
quantitative  process  to  assess  the  alternatives. 

4.  Document  and  review  the  AOA  process:  includes  best  practices  that 
would  be  applied  throughout  the  AOA  process,  such  as  documenting 


^he  best  practices  listed  in  this  appendix  are  an  update  of,  and  supersede,  the  initial  set 
of  24  best  practices  listed  in  GAO-15-37. 
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all  steps  taken  to  initialize,  identify,  and  analyze  alternatives  and  to 
select  a  preferred  alternative  in  a  single  document. 

5.  Select  a  preferred  alternative:  includes  a  best  practice  that  is  applied 
by  the  decision  maker  to  compare  alternatives  and  to  select  a 
preferred  alternative. 

The  five  phases  address  different  themes  of  analysis  necessary  to 
complete  the  AOA  process  and  comprise  the  beginning  of  the  AOA 
process  (defining  the  mission  needs  and  functional  requirements)  through 
the  final  step  of  the  AOA  process  (selecting  a  preferred  alternative). 

There  are  three  key  entities  that  are  involved  in  the  AOA  process:  the 
customer,  the  decision  maker,  and  the  AOA  team.  The  customer  refers  to 
the  program  office,  service,  or  agency  that  identifies  a  mission  need  (e.g. 
a  credible  gap  between  current  capabilities  and  those  required  to  meet 
the  goals  articulated  in  the  strategic  plan).  The  decision  maker  is  the 
person  or  entity  that  signs  off  on  the  final  decision  and  analysis 
documented  by  the  AOA  report.  The  decision  maker  refers  to  the  program 
manager  (or  alternate  authority  figure  identified  early  in  the  AOA  process) 
who  will  select  the  preferred  alternative  based  on  the  established 
selection  criteria.  The  AOA  team  is  the  group  of  subject  matter  experts 
who  are  involved  in  the  day-to-day  work  of  the  AOA  process  and  work  to 
develop  the  analysis  that  is  the  foundation  of  the  AOA  process. 

Conforming  to  the  22  best  practices  helps  ensure  that  the  preferred 
alternative  selected  is  the  one  that  best  meets  the  agency’s  mission 
needs.  Not  conforming  to  the  best  practices  may  lead  to  an  unreliable 
AOA,  and  the  customer  will  not  have  assurance  that  the  preferred 
alternative  best  meets  the  mission  needs.  Table  5  shows  the  22  best 
practices  and  the  five  phases: 
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Table  5:  Best  Practices  for  the  Analysis  of  Alternatives  (AOA)  Process 


Phase  I.  Initialize  the  AOA  process 

1.  Define  mission  need 

Definition:  The  customer  defines  the  mission  needs  (i.e.,  a  credible  gap  between  current  capabilities  and  those  required  to  meet  the 
goals  articulated  in  the  strategic  plan)  without  a  predetermined  solution.  To  ensure  that  the  AOA  process  does  not  favor  one  solution 
over  another,  the  AOA  is  conducted  before  design  and  development  of  the  required  capabilities.  The  customer  decides  at  which  level 
of  design  completion  an  AOA  should  be  performed;  with  the  understanding  that  the  more  complete  the  design,  the  more  information  is 
available  to  support  a  robust  analysis  and  to  select  a  preferred  alternative  that  best  meets  the  mission  need. 

Effect:  Allowing  mission  needs  to  be  defined  in  solution-specific  terms  creates  a  potential  bias  and  could  invalidate  the  analysis. 

2.  Define  functional  requirements 

Definition:  The  customer  defines  functional  requirements  (i.e.  the  general  parameters  that  the  selected  alternative  must  have  to 
address  the  mission  need)  based  on  the  mission  need  without  a  predetermined  solution.  The  customer  defines  the  capabilities  that 
the  AOA  process  seeks  to  refine  through  characterized  gaps  between  capabilities  in  the  current  environment  and  the  capabilities 
required  to  meet  the  stated  objectives  for  the  future  environment.  These  functional  requirements  are  realistic,  organized,  clear, 
prioritized,  and  traceable.  It  is  advisable  that  functional  requirements  be  set  early  in  the  AOA  process  and  agreed  upon  by  all 
stakeholders. 

Effect:  The  AOA  process  is  tied  to  the  identified  mission  needs.  Setting  functional  requirements  to  a  standard  other  than  mission 
needs  allows  bias  to  enter  the  study  because  the  requirements  might  then  reflect  arbitrary  measures.  Additionally,  requirements  not 
tied  to  mission  needs  make  it  difficult  to  quantify  the  benefits  of  each  alternative  relative  to  what  is  required  and  make  it  challenging  for 
decision  makers  to  assess  which  capability  gaps  will  be  met  for  each  alternative. 

3.  Develop  AOA  time  frame 

Definition:  The  customer  provides  the  team  conducting  the  analysis  enough  time  to  complete  the  AOA  in  order  to  ensure  a  robust  and 
complete  analysis.  Since  an  AOA  process  requires  a  large  team  with  many  diverse  resources  and  expertise,  the  process  requires 
sufficient  time  to  be  accomplished  thoroughly.  A  detailed  schedule  is  developed  prior  to  starting  the  AOA  process.  The  duration  of  the 
AOA  process  depends  on  the  number  of  viable  alternatives  and  availability  of  the  team  members.  The  time  frame  is  tailored  for  the 
type  of  system  to  be  analyzed  and  ensures  that  there  is  adequate  time  to  accomplish  all  of  the  AOA  process  steps  robustly. 

Effect:  The  AOA  process  identifies  and  thoroughly  analyzes  a  comprehensive  range  of  alternatives.  Recommending  an  alternative 
without  adequate  time  to  perform  the  analysis  is  a  contributing  factor  to  high-dollar  acquisitions  that  have  significantly  overrun  both 
cost  and  schedule  while  falling  short  of  expected  performance. 

4.  Establish  AOA  team 

Definition:  After  the  customer  establishes  the  need  for  the  AOA  in  steps  1  through  3,  a  diverse  AOA  team  is  established  to  develop  the 
AOA.  This  team  consists  of  members  with  a  variety  of  necessary  skill  sets,  specific  knowledge,  and  abilities  to  successfully  execute 
the  study.  For  example,  the  AOA  team  includes  individuals  with  skills  and  experience  in  the  following  areas:  program  management, 
federal  contracting,  cost  estimating,  risk  management,  sustainability,  scheduling,  operations,  technology,  earned  value  management, 
budget  analysis,  and  any  other  necessary  areas  of  expertise. 

Effect:  An  AOA  process  includes  a  diverse  group  of  subject  matter  experts  to  perform  the  analysis.  Since  each  subject  matter  expert 
brings  their  knowledge  to  the  team,  without  the  appropriate  expertise  on  the  team,  errors  in  the  results  could  occur  and  gaps  in  the 
analysis  could  be  created,  causing  the  AOA’s  completion  to  be  delayed  as  more  subject  matter  experts  are  identified  and  tasked  to 
work  as  part  of  the  AOA  process. 

5.  Define  selection  criteria 

Definition:  The  AOA  team  or  the  decision  maker  defines  selection  criteria  based  on  the  mission  need.  The  defined  criteria  are  based 
on  mission  needs  and  are  independent  of  a  particular  capital  asset  or  technological  solution.  The  selection  criteria  are  defined  based 
on  the  mission  need  prior  to  starting  the  analysis. 

Effect:  It  is  essential  that  the  selection  criteria  be  based  on  the  mission  needs.  If  there  are  no  preset  criteria  based  on  documented 
requirements,  bias  can  enter  the  AOA  process  and  prevent  the  decision  maker  from  forming  an  impartial  and  unbiased  decision. 
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6.  Weight  selection  criteria 

Definition:  The  AOA  team  or  the  decision  maker  weights  the  selection  criteria  to  reflect  the  relative  importance  of  each  criterion.  While 
the  selection  criteria  are  ranked  in  importance,  the  alternatives  are  based  on  trade-offs  between  costs,  operational  effectiveness, 
risks,  schedules,  flexibility,  and  other  factors  identified  by  the  team  or  the  decision  maker. 

Effect:  An  unjustified  weighting  method  can  oversimplify  the  results  and  potentially  mask  important  information,  leading  to  an 
uninformed  decision. 

7.  Develop  AOA  process  plan 

Definition:  The  AOA  team  creates  a  plan  to  include  proposed  methodologies  for  identifying,  analyzing,  and  selecting  alternatives  prior 
to  beginning  the  AOA  process.  This  plan  establishes  the  critical  questions  to  be  explored,  the  selection  criteria,  the  basis  of  estimates, 
and  measures  that  are  used  to  rate,  rank,  and  decide  among  the  alternatives.  Additionally,  the  plan  includes  the  criteria  used  to 
determine  each  alternative’s  viability.  A  road  map  and  standard  work  breakdown  structure  are  used  to  compare  the  alternatives  with 
the  baseline  and  with  each  other. 

Effect:  The  functional  requirements  and  selection  criteria  are  identified  prior  to  the  beginning  of  the  analysis.  If  criteria  to  select  the 
preferred  alternative  are  established  after  the  analysis  has  begun,  bias  may  influence  the  study’s  results.  Furthermore,  if  planned 
methodologies  for  the  remaining  phases  of  the  AOA  study  are  not  established,  the  risk  of  applying  poor  methodologies  as  part  of  the 
AOA  analysis  increases. 

Phase  II.  Identify  alternatives 

8.  Develop  list  of  alternatives 

Definition:  The  AOA  team  identifies  and  considers  a  diverse  range  of  alternatives  to  meet  the  mission  need.  To  fully  address  the 
capability  gaps  between  the  current  environment  and  the  stated  objectives  for  the  future  environment,  market  surveillance  and  market 
research  is  performed  to  develop  as  many  alternative  solutions  as  possible  for  examination.  Alternatives  are  mutually  exclusive,  that 
is,  the  success  of  one  alternative  does  not  rely  upon  the  success  of  another. 

Effect:  An  AOA  process  encompasses  numerous  alternatives  in  order  to  ensure  that  the  study  provides  a  broad  view  of  the  issue.  If 
the  AOA  team  does  not  perform  thorough  research  to  capture  diverse  alternatives,  the  optimal  alternative  could  be  overlooked  and 
invalidate  the  AOA’s  results  and  bias  the  process. 

9.  Describe  alternatives 

Definition:  The  AOA  team  describes  alternatives  in  sufficient  detail  to  allow  for  robust  analysis.  All  alternatives’  scope  is  described  in 
terms  of  functional  requirements.  This  description  is  detailed  enough  to  support  the  viability,  cost,  and  benefit/effectiveness  analyses. 
Effect:  Documentation  is  essential  for  validating  the  AOA  process  and  defending  its  conclusions.  Unless  the  AOA  team  adequately 
describes  and  documents  the  alternatives,  the  analysis  will  not  provide  sufficient  detail  to  allow  for  valid  cost-benefit  estimates  and  will 
not  be  credible. 

10.  Include  baseline  alternative 

Definition:  The  AOA  team  includes  one  alternative  to  represent  the  status  quo  to  provide  a  basis  of  comparison  among  alternatives.  It 
is  critical  for  the  AOA  team  to  first  understand  the  status  quo,  which  represents  the  existing  capability’s  baseline  where  no  action  is 
taken,  before  comparing  alternatives.  The  baseline  is  well  documented  as  an  alternative  in  the  study  and  is  used  to  represent  the 
current  capabilities  and  also  for  explicit  comparison  later  in  the  study. 

Effect:  It  is  essential  that  the  AOA  process  compare  the  current  environment  with  the  possible  future  environment.  If  no  status  quo  is 
examined,  then  there  is  no  benchmark  for  comparison,  allowing  arbitrary  comparisons  between  alternatives  and  hindering  the 
credibility  of  the  study. 
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1 1 .  Assess  alternatives’  viability 

Definition:  The  AOA  team  screens  the  list  of  alternatives  to  eliminate  those  alternatives  that  are  not  viable,  and  it  documents  the 
reasons  for  eliminating  any  alternatives.  All  alternatives  are  examined  using  predetermined  qualitative  technical  and  operational 
factors  to  determine  their  viability.  Only  those  alternatives  found  viable  are  examined  fully  in  the  AOA  process.  However,  all 
assumptions  regarding  the  alternatives’  viable  and  nonviable  status  are  fully  documented,  including  reasons  that  an  alternative  is  not 
viable,  in  order  to  justify  the  recommendation.  Additionally,  viable  alternatives  that  are  not  affordable  within  the  projected  available 
budget  are  dropped  from  final  consideration. 

Effect:  Not  eliminating  alternatives  based  on  viability  could  needlessly  extend  the  study’s  duration  and  burden  the  AOA  team  or  lead  to 
the  selection  of  a  technically  nonviable  alternative.  Furthermore,  unless  the  AOA  team  considers  affordability  as  part  of  the  final 
recommendation,  an  alternative  that  is  not  feasible  based  on  the  current  fiscal  environment  could  be  selected.  Documenting  the 
alternatives  that  are  deemed  nonviable  is  important  so  that  decision  makers  can  clearly  see  why  those  alternatives  are  not  considered 
for  further  analysis. 

Phase  III.  Analyze  alternatives 

12.  Identify  significant  risks  and  mitigation  strategies 

Definition:  The  AOA  team  identifies  and  documents  the  significant  risks  and  mitigation  strategies  for  each  alternative.  Risks  are 
ranked  in  terms  of  significance  to  mission  needs  and  functional  requirements.  All  risks  are  documented  for  each  alternative  along  with 
any  overarching  or  alternative  specific  mitigation  strategies.  Schedule  risk,  cost  risk,  technical  feasibility,  risk  of  technical 
obsolescence,  dependencies  between  a  new  project  and  other  projects  or  systems,  procurement  and  contract  risk,  and  resources 
risks  are  examined. 

Effect:  Since  AOA  processes  typically  occur  early  in  the  acquisition  process,  risk  is  inherently  a  part  of  every  alternative.  Not 
documenting  the  risks  and  related  mitigation  strategies  for  each  alternative  prevents  decision  makers  from  performing  a  meaningful 
trade-off  analysis  necessary  to  choose  a  recommended  alternative. 

13.  Determine  and  quantify  benefits/effectiveness 

Definition:  The  AOA  team  uses  a  standard  process  to  document  the  benefits  and  effectiveness  of  each  alternative.  The  AOA  team 
drafts  a  metric  framework  that  details  the  methods  used  to  evaluate  and  quantify  the  measures  of  effectiveness  and  measures  of 
performance  for  all  mission  needs.  The  AOA  team  quantifies  the  benefits  and  effectiveness  of  each  alternative  over  the  alternative’s 
full  life-cycle,  if  possible.  Just  as  costs  cover  the  entire  life-cycle  for  each  alternative,  the  benefits  and  effectiveness  measures  cover 
each  alternative’s  life-cycle,  if  possible,  in  order  to  determine  each  alternative’s  Net  Present  Value — i.e.,  the  discounted  value  of 
expected  benefits  minus  the  discounted  value  of  expected  costs.  In  cases  where  the  means  to  monetize  a  benefit  are  too  vague  (for 
example,  intangibles  like  scientific  knowledge),  the  AOA  team  treats  those  benefits  as  strategic  technical  benefits  and  uses  scalability 
assessments  to  quantify  those  benefits  so  that  they  are  compared  across  all  viable  alternatives.  In  a  situation  where  benefits  cannot 
be  quantified,  the  AOA  team  explains  why  this  is  the  case  as  part  of  their  analysis. 

Effect:  Determining  a  standard  process  to  quantify  benefits  is  an  essential  part  of  the  AOA  process.  If  the  AOA  team  does  not  clearly 
establish  criteria  against  which  to  measure  all  alternatives,  bias  is  introduced  to  the  study.  Additionally,  if  the  AOA  team  does  not 
examine  effectiveness  over  the  entire  life-cycle,  decision  makers  cannot  see  the  complete  picture  and  are  prevented  from  making  an 
informed  decision. 

14.  Tie  benefits/effectiveness  to  mission  need 

Definition:  The  AOA  team  explains  how  each  measure  of  effectiveness  supports  the  mission  need.  The  AOA  team  shows  how  the 
measures  of  effectiveness  describe  the  way  the  current  environment  is  expected  to  evolve  to  meet  the  desired  environment;  the  team 
also  shows  how  the  measures  are  tied  to  specific  mission  needs  and  functional  requirements.  This  is  the  hierarchy  that  connects  the 
overarching  requirements  to  the  data  that  are  needed. 

Effect:  Unless  the  AOA  team  thoroughly  documents  how  the  measures  of  effectiveness  relate  to  specific  mission  needs  and  functional 
requirements,  decision  makers  will  not  have  proper  insight  into  the  impact  of  each  alternative. 
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15.  Develop  life-cycle  cost  estimates 

Definition:  The  AOA  team  develops  a  life-cycle  cost  estimate  for  each  alternative,  including  all  costs  from  inception  of  the  project 
through  design,  development,  deployment,  operation,  maintenance,  and  disposal.  The  AOA  team  includes  a  cost  expert  who  is 
responsible  for  development  of  a  comprehensive,  well-documented,  accurate,  and  credible  cost  estimate  for  each  viable  alternative  in 
the  study.  The  life-cycle  cost  estimate  for  each  alternative  follows  the  GAO  12-step  guide  and  uses  a  common  cost  element  structure 
for  all  alternatives  and  includes  all  costs  for  each  alternative.3  Costs  that  are  the  same  across  the  alternatives  (for  example,  training 
costs)  are  included  so  that  decision  makers  can  compare  the  total  cost  rather  than  just  the  portion  of  costs  that  varies  across  all  viable 
alternatives.  The  AOA  team  expresses  the  life-cycle  cost  estimate  in  present  value  terms  and  explains  why  it  chose  the  specific 
discount  rate  used.  The  AOA  team  ensures  that  economic  changes,  such  as  inflation  and  the  discount  rate,  are  properly  applied, 
realistically  reflected,  and  documented  in  the  life  cycle  cost  estimate  for  all  alternatives.  Furthermore,  the  present  value  of  the  estimate 
reflects  the  time  value  of  money;  the  concept  that  a  dollar  today  can  be  invested  and  earn  interest. 

Effect:  A  life-cycle  cost  estimate  that  is  incomplete  (i.e.  does  not  include  all  phases  of  an  alternative’s  life-cycle)  does  not  provide  an 
accurate  and  complete  view  of  the  alternatives’  costs.  Without  a  full  accounting  of  life-cycle  costs,  decision  makers  will  not  have  a 
complete  picture  of  the  costs  for  each  alternative  and  will  have  difficulty  comparing  the  alternatives  because  comparisons  may  not  be 
based  on  accurate  information.  Additionally,  applying  a  discount  rate  is  an  important  step  in  cost  estimating  because  all  cost  data 
must  be  expressed  in  like  terms  for  comparison.  Unless  the  AOA  team  properly  normalizes  costs  to  a  common  standard,  any 
comparison  would  not  be  accurate,  and  any  recommendations  resulting  from  the  flawed  analysis  would  be  negated.  Properly 
normalizing  costs  is  particularly  important  if  various  alternatives  have  different  life-cycles. 

16.  Include  a  confidence  interval  or  range  for  life-cycle  cost  estimates 

Definition:  The  AOA  team  presents  the  life-cycle  cost  estimate  for  each  alternative  with  a  confidence  interval  or  range,  and  not  solely 
as  a  point  estimate.  To  document  the  level  of  risk  associated  with  the  point  estimate  for  each  viable  alternative,  the  confidence  interval 
is  included  as  part  of  the  life-cycle  cost  estimates  for  each  viable  alternative  (in  accordance  with  GAO  Cost  Estimating  Best  Practice 
#9,  risk  and  uncertainty  analysis).15  Decision  makers  must  have  access  to  the  confidence  interval  associated  with  the  point  estimates 
for  all  viable  alternatives  in  order  to  make  informed  decisions.  Additionally,  the  AOA  team  uses  a  consistent  method  of  comparing 
alternatives  in  order  to  present  a  comparable  view  of  the  risk  associated  with  each  alternative.  For  example,  the  comparison  can  be 
based  on  an  established  dollar  value  across  alternatives  (in  order  to  observe  the  confidence  level  for  each  alternative  at  that  dollar 
value).  Alternatively,  the  comparison  can  be  based  on  a  predetermined  confidence  level  across  alternatives  (in  order  to  observe  the 
dollar  value  associated  with  that  confidence  level  for  each  alternative). 

Effect:  For  decision  makers  to  make  an  informed  decision,  the  alternatives’  life-cycle  cost  estimates  must  reflect  the  degree  of 
uncertainty.  Having  a  range  of  costs  around  a  point  estimate  is  useful  because  it  conveys  a  level  of  confidence  for  each  alternative  to 
achieve  a  most  likely  cost.  Without  cost  risk  and  uncertainty  analysis  the  life-cycle  cost  estimates  for  the  viable  alternatives  are  not 
credible. 

17.  Perform  sensitivity  analysis 

Definition:  The  AOA  team  tests  and  documents  the  sensitivity  of  the  cost  and  benefit  and  effectiveness  estimates  for  each  alternative 
to  risks  and  changes  in  key  assumptions.  Major  outcomes  and  assumptions  are  varied  in  order  to  determine  each  alternative’s 
sensitivity  to  changes  in  key  assumptions.  This  analysis  is  performed  in  order  to  rank  the  key  drivers  that  could  influence  the  cost  and 
benefit  estimates  based  on  how  they  affect  the  final  results  for  each  alternative.  Each  alternative  includes  both  a  sensitivity  and  risk 
and  uncertainty  analysis  that  identifies  a  range  of  possible  costs  based  on  varying  key  assumptions,  parameters,  and  data  inputs.  As 
explained  in  best  practice  #16,  life-cycle  cost  estimates  are  adjusted  to  account  for  risk  and  sensitivity  analyses. 

Effect:  Failing  to  conduct  a  sensitivity  analysis  to  identify  the  uncertainties  associated  with  different  assumptions  increases  the  chance 
the  AOA  team  will  recommend  an  alternative  without  an  understanding  of  the  full  impacts  on  life-cycle  costs,  which  could  lead  to  cost 
and  schedule  overruns. 
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Phase  IV.  Document  and  review  the  AOA  process 

18.  Document  AOA  process  in  a  single  document 

Definition:  The  AOA  team  documents  all  steps  taken  to  identify,  analyze,  and  select  alternatives  in  a  single  document.  This  document 
clearly  states  the  preferred  alternative  and  provides  the  detailed  rationale  for  the  recommendation  based  on  analytic  results.  The 
report  includes  sections  detailing  the  steps  taken  to  initialize  the  AOA  process,  and  to  identify,  analyze,  and  select  alternatives.  For 
example,  one  section  lists  the  overall  selection  criteria  and  rationale  for  nonviable  or  viable  ratings  for  alternatives,  assumptions  for 
each  alternative,  risk  drivers  and  mitigation  techniques,  analysis  of  the  costs  and  benefits  associated  with  each  alternative,  and  the 
trade-offs  between  costs,  benefits,  and  risks. 

Effect:  Documentation  is  essential  for  validating  and  defending  the  AOA  process.  Without  clear  reports  that  compile  all  information, 
including  standards  used  to  rate  and  perform  the  analysis,  the  study’s  credibility  could  suffer  because  the  documentation  does  not 
explain  the  rationale  for  the  methodology  or  the  calculations  underlying  the  analysis.  Having  all  the  information  related  to  all  best 
practices  of  the  AOA  process  in  one  single  document  also  makes  it  easier  for  an  independent  reviewer  to  assess  the  AOA  process. 

19.  Document  assumptions  and  constraints 

Definition:  The  AOA  team  documents  and  justifies  all  assumptions  and  constraints  used  in  the  AOA  process.  Assumptions  and 
constraints  help  to  scope  the  AOA.  Assumptions  are  explicit  statements  used  to  specify  precisely  the  environment  to  which  the 
analysis  applies,  while  constraints  are  requirements  or  other  factors  that  cannot  be  changed  to  achieve  a  more  beneficial  approach. 
Both  assumptions  and  constraints  are  detailed  and  justified  for  each  alternative  in  the  AOA  plan. 

Effect:  Without  documented  and  justified  assumptions  and  constraints  it  will  be  difficult  for  decision  makers  to  evaluate  the 
alternatives. 

20.  Ensure  AOA  process  is  impartial 

Definition:  The  AOA  team  conducts  the  analysis  without  a  predetermined  solution.  The  AOA  process  informs  the  decision-making 
process  rather  than  reflecting  the  validation  of  a  predetermined  solution.  The  AOA  process  is  an  unbiased  inquiry  into  the  costs, 
benefits,  and  capabilities  of  all  alternatives. 

Effect:  An  AOA  process  is  not  considered  valid  if  it  is  biased.  Performing  a  study  with  a  predetermined  solution  distorts  the  results. 
The  validity  of  the  analysis  is  affected  if  bias  is  introduced  to  the  inputs. 

21.  Perform  independent  review 

Definition:  An  entity  independent  of  the  AOA  process  reviews  the  extent  to  which  all  best  practices  are  followed.  The  AOA  process  is 
completed  with  enough  thoroughness  to  ensure  that  an  independent  organization  outside  of  the  project’s  chain  of  command  can 
review  the  AOA  documentation  and  clearly  understand  the  process  and  rationale  that  led  to  the  selection  of  the  recommended 
alternative.  Part  of  the  documentation  includes  approval  and  review  from  an  office  outside  of  the  one  that  asked  for  or  performed  the 
AOA  process.  For  certain  projects,  in  addition  to  an  independent  review  at  the  end  of  the  AOA  process,  additional  independent 
reviews  are  necessary  at  earlier  stages  of  the  process,  such  as  reviews  of  the  AOA  process  plan  of  the  identification  of  viable 
alternatives.  While  early  reviews  are  not  a  substitute  for  the  independent  review  conducted  at  the  end  of  the  AOA  process,  they  help 
ensure  that  bias  is  not  added  through  the  course  of  the  AOA  process. 

Effect:  An  independent  review  is  one  of  the  most  reliable  means  to  validate  an  AOA  process.  Without  an  independent  review,  the 
results  are  more  likely  to  include  organizational  bias  or  lack  the  thoroughness  needed  to  ensure  that  a  preferred  solution  is  chosen 
and  not  a  favored  solution. 

Phase  V.  Select  a  preferred  alternative 

22.  Compare  Alternatives 

Definition:  The  AOA  team  or  the  decision  maker  compares  the  alternatives  using  net  present  value,  if  possible,  to  select  a  preferred 
alternative.  Net  present  value  can  be  negative  if  discounted  costs  are  greater  than  discounted  benefits.  Net  present  value  is  the 
standard  criteria  used  when  deciding  whether  an  alternative  can  be  justified  based  on  economic  principles.  In  some  cases,  net 
present  value  cannot  be  used,  such  as  when  quantifying  benefits  is  not  possible.  In  these  cases,  the  AOA  team  documents  why  net 
present  value  cannot  be  used.  Furthermore,  if  net  present  value  is  not  used  to  differentiate  among  alternatives,  the  AOA  team  should 
document  why  net  present  value  is  not  used,  and  describe  the  other  method  that  is  used  to  differentiate,  and  explain  why  that  method 
has  been  applied. 

Effect:  Comparing  items  that  have  not  been  discounted  (or  normalized)  does  not  allow  for  time  series  comparisons  since  alternatives 
may  have  different  life  cycles  or  different  costs  and  benefits. 


Source:  GAO-16-22  |  GAO-16-853 


Page  61 


GAO-16-853  Joint  Intelligence  Analysis  Complex 


Appendix  III:  Best  Practices  for  the  Analysis  of 
Alternatives  Process 


aGAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing 
Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  Mar.  2,  2009). 

bGAO-09-3SP. 

Some  best  practices  included  in  a  phase  can  take  place  concurrently  and 
do  not  have  to  follow  the  order  presented  in  table  5.  The  phases  should 
occur  in  sequence  to  prevent  bias  from  entering  the  analysis  and  adding 
risk  that  the  AOA  team  will  analyze  alternatives  that  have  not  been 
defined.  However,  the  document  and  review  phase  can  be  done  at  any 
stage  throughout  the  AOA  process.  For  example,  best  practice  5  (define 
selection  criteria)  can  be  done  at  the  same  time  as  best  practice  6  (weight 
selection  criteria).  On  the  other  hand,  best  practice  20  (ensure  AOA 
process  is  impartial)  can  be  done  at  the  end  of  every  step  or  every  phase 
to  ensure  the  impartiality  of  the  AOA  as  it  progresses.  The  best  practices 
represent  an  overall  process  that  results  in  a  reliable  AOA  that  can  be 
easily  and  clearly  traced,  replicated,  and  updated.  Figure  3  shows  the 
AOA  process  and  how  the  steps  in  each  phase  are  interrelated. 
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Figure  3:  Analysis  of  Alternatives  (AOA)  Process  Chart 


Document  and  review 


AOA  Analysis  of  alternatives 
LCCE  Life-cycle  cost  estimate 

Source:  GAO.  |  GAO-16-853 

Note:  The  figure  displays  the  AOA  process  by  phase  and  step.  The  “Initialize,  Identify,  Analyze,  and 
Select”  phases  should  be  done  in  order,  but  the  “Document  and  Review”  phase  can  be  done 
throughout  the  AOA  process.  The  arrows  indicate  that  the  “Document  and  Review”  phase  is  related  to 
the  other  four  phases.  Within  each  phase,  there  are  steps  that  can  be  done  concurrently  rather  than 
consecutively.  The  concurrent  steps  are  grouped  together  in  dark  blue  boxes.  Furthermore,  there  are 
steps  in  later  phases  that  are  related  to  steps  in  earlier  phases;  these  are  connected  with  a  two-way 
arrow. 
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Above,  in  our  discussion  of  the  extent  to  which  DOD’s  AOA  process  met 
best  practices  for  such  processes,  we  presented  our  analysis  for  certain 
best  practices.  In  table  6,  we  summarize  our  analysis  of  DOD’s  AOA 
process  for  the  remaining  best  practices. 


Table  6:  GAO  Analysis  of  the  Department  of  Defense’s  (DOD)  Analysis  of  Alternatives  (AOA)  Process  by  Best  Practice 

Best  practices  for  the  AOA  process 

Summary  of  GAO’s  analysis 

1.  Initialize  the  AOA  process 

1.  Define  mission  need 

Score  5  -  Fully  Met 

DOD’s  AOA  body  of  work  includes  detailed  documents  that  laid  out  both  the  shortfalls  of 
current  Joint  Intelligence  Analysis  and  Production  Complex  (JIAO)  facilities  and  the 
specifications  for  a  new  JIAC  facility,  based  on  the  mission  needs  of  the  JIAC’s 
organizations. 

2.  Define  functional  requirements 

Score  5  -  Fully  Met 

The  AOA  body  of  work  provides  the  general  parameters  that  the  selected  alternative  must 
have  to  address  the  mission  need.  For  example,  in  a  requirements  document,  the  AOA 
team  discusses  how  established  standards  for  office  space — from  Defense  Intelligence 
Agency  and  Army  policy — drive  the  JIAC  facilities’  office  space  requirements.  Also,  the 

AOA  body  of  work  defines  the  gaps  between  desired  capabilities  in  the  current 
environment.  For  instance,  JIAC  consolidation  budget  justification  documents  state  that 
intelligence  mission  growth  at  RAF  Molesworth  of  over  500  percent  since  1991  has 
resulted  in  a  severe  shortfall  of  intelligence  spaces,  resulting  in  intelligence  missions 
being  housed  in  undersized  facilities. 

3.  Develop  AOA  timeframe 

Score  3  -  Partially  Met 

During  the  team’s  discussion  with  DOD  officials,  they  provided  dates  for  several  key 
events  during  the  AOA  process  and  indicated  that  there  was  a  rough  timeline  in  place  that 
the  department  needed  to  follow  to  meet  the  dates  within  the  DOD  budget  process  and 
consolidate  as  soon  as  possible,  given  the  degraded  JIAC  facilities’  condition  and  impact 
on  missions  located  at  RAF  Molesworth.  However,  the  body  of  work  does  not  provide 
specific  evidence  that  a  timeline  for  the  analysis  of  alternatives  process  was  established 
prior  to  starting  the  AOA  process,  which  is  a  requirement  of  the  process. 

4.  Establish  AOA  team 

Score  3  -  Partially  Met 

There  is  evidence  in  the  AOA  body  of  work  that  the  collection  of  subject  matter  experts 
involved  in  the  AOA  process  (i.e. ,  the  AOA  team)  consisted  of  members  with  certain 
necessary  skill  sets,  specific  knowledge,  and  abilities  to  successfully  execute  the  study. 

For  example,  the  201 1  preliminary  AOA  summary  states  that  EUCOM,  AFRICOM,  the 
Office  of  the  Secretary  of  Defense,  and  military  service  stakeholders  worked  together  to 
develop  a  recapitalization  plan  predicated  on  a  thoughtful,  risk-based  analysis  of 
alternatives.  Given  that  these  DOD  organizations  are  key  stakeholders  in  the  operation  of 
the  JIAC  and  execution  of  its  mission,  the  subject  matter  expertise  of  their  personnel  is 
relevant  to  AOA  decision  making.  However,  the  body  of  work  does  not  include  specific 
documentation  listing  the  specific  subject  matter  expertise  required  for  the  AOA  process 
or  the  specific  experts  who  provided  that  expertise  during  the  process.  Thus,  DOD  cannot 
demonstrate  that  its  AOA  team  contained  all  the  necessary  subject  matter  experts. 

5.  Define  selection  criteria 

Score  5  -  Fully  Met 

DOD’s  AOA  body  of  work  provides  evidence  the  department’s  AOA  team  defined  the  five 
AOA  criteria  DOD  based  on  mission  needs  of  the  JIAC.  For  example,  the  201 1 
preliminary  AOA  summary  defines  the  second  criterion — ’’Impact  on  Bilateral  and 
Multinational  Intelligence  Collaboration” —  in  the  context  of  the  JIAC’s  mission,  explaining 
that,  given  the  important  role  collaboration  plays  in  its  operations,  options  were  assessed 
for  their  potential  impacts  on  international  intelligence  partnerships. 
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Best  Practices  and  Four  Characteristics,  With 

GAO’s  Evaluation  of  the  Department  of 

Defense’s  (DOD)  AOA  Process 

Best  practices  for  the  AOA  process 

Summary  of  GAO’s  analysis 

6.  Weight  selection  criteria 

Score  3  -  Partially  Met 

In  the  AOA  body  of  work,  the  AOA  team  identifies  the  first  two  criteria  as  critical,  which  is 
a  form  of  weighting.  Because  four  of  the  five  AOA  criteria  used  by  the  team  were 
qualitative — and  two  criteria  are  weighted  via  their  label  of  “critical” — the  AOA  team 
partially  met  the  requirements  of  this  best  practice.  However,  DOD  provides  no  indication 
of  the  relative  weight  of  the  remaining  three  criteria.  This  lack  of  weighting  makes  it 
unclear  the  degree  to  which  the  AOA  team  considered  alternatives’  costs  and  savings  in 
the  AOA  process. 

7.  Develop  AOA  process  plan 

Score  2  -  Minimally  Met 

In  the  AOA  body  of  work,  there  is  evidence  that  the  AOA  team  took  actions  that  could 
reasonably  be  expected  to  be  part  of  an  AOA  process  plan.  For  example,  the  201 1 
preliminary  AOA  summary  generally  describes  the  process  followed  by  the  AOA  team. 
However,  the  body  of  work  does  not  provide  evidence  of  a  process  plan  of  the  type 
required  by  this  best  practice.  Specifically,  there  is  no  evidence  that  DOD  developed  a 
document  that  establishes  the  critical  questions  to  be  explored;  the  selection  criteria;  the 
basis  of  estimates;  measures  that  are  used  to  rate,  rank,  and  decide  among  the 
alternatives;  the  criteria  used  to  determine  each  alternative’s  viability;  or  a  road  map  and 
standard  work  breakdown  structure  used  to  compare  the  alternatives  with  the  baseline 
and  with  each  other. 

II.  Identify  alternatives 

8.  Develop  list  of  alternatives 

Score  4  -  Substantially  Met 

The  AOA  team  identified  and  considered  a  wide  range  of  alternatives  to  meet  the  mission 
need.  According  to  DOD  officials,  the  department  considered  up  to  16  alternatives. 

9.  Describe  alternatives 

Score  2  -  Minimally  Met 

Although  the  AOA  team  identified  and  considered  a  wide  range  of  alternatives  to  meet  the 
mission  need,  DOD’s  AOA  body  of  work  does  not  provide  evidence  that  the  AOA  team 
met  the  requirements  for  describing  all  of  the  alternatives.  For  example,  in  the  201 1 
preliminary  AOA  summary,  the  AOA  team  described  the  alternatives  in  only  basic  terms, 
using  several  words  or  one  sentence. 

10.  Include  baseline  alternative 

Score  4  -  Substantially  Met 

In  the  AOA  body  of  work,  the  AOA  team  includes  one  alternative  referred  to  as  “Defer 
capitalization  (maintain  status  quo),”  which  represents  the  status  quo  and  provides  a  basis 
of  comparison  among  alternatives.  This  baseline  was  well  documented.  For  example, 

DOD  1391  forms — used  to  provide  Congress  with  information  on  military  construction 
projects — include  information  on  the  current  JIAC  facilities’  capabilities,  explaining  that 
they  suffer  from  a  severe  shortfall  of  intelligence  spaces.  Further,  the  AOA  body  of  work 
also  included  detailed  requirements  documents  that  laid  out  the  shortfalls  of  current  JIAC 
facilities.  However,  the  baseline  alternative,  like  all  other  alternatives,  does  not  have  a  full 
life  cycle  cost  analysis. 

1 1 .  Assess  alternatives’  viability 

Score  2  -  Minimally  Met 

According  to  DOD  officials,  the  AOA  team  determined  alternatives’  viability  by  examining 
the  alternatives  using  the  department’s  own  five  AOA  criteria.  However,  the  DOD  AOA 
body  of  work  does  not  contain  documentation  indicating  that  the  AOA  team  met  several 
requirements  of  our  best  practice.  For  example,  the  body  of  work  does  not  contain  a 
documented  description  of  the  process  that  the  AOA  team  used  to  add  and  remove 
alternatives  over  time. 

III.  Analyze  alternatives 

12.  Identify  significant  risks  and 
mitigation  strategies 

Score  3  -  Partially  Met 

DOD’s  AOA  body  of  work  provides  evidence  that  the  AOA  team  generally  identified  risks 
and  mitigation  strategies,  organizing  its  discussion  of  risk  and  mitigation  by  criterion  (as 
opposed  to  organizing  by  alternative).  For  example,  in  a  201 1  preliminary  AOA  summary, 
the  team  identifies  several  risks  and  mitigation  strategies  for  meeting  the  criterion  “Impact 
on  Intelligence  Operations.”  However,  the  body  of  work  does  not  provide  evidence  that 
risks  are  ranked;  risks  and  strategies  are  documented  for  each  alternative;  or  that  the 
seven  specific  types  of  risks  required  to  be  identified  by  this  best  practice  are  discussed. 
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Best  practices  for  the  AOA  process 

Summary  of  GAO’s  analysis 

13.  Determine  and  quantify 
benefits/effectiveness 

Score  2  -  Minimally  Met 

The  AOA  body  of  work  provides  some  evidence  that  the  AOA  team  partially  met  certain 
requirements  for  this  best  practice.  Specifically,  the  team  seems  to  have  used  a  standard 
process  to  document  the  benefits  and  effectiveness  of  the  alternatives.  The  201 1 
preliminary  AOA  summary  includes  a  table  in  which  alternatives  were  assessed  and 
compared  using  a  relative  stoplight  chart  “scoring”  construction.  However,  this  table  does 
not  include  each  alternative.  Instead,  it  includes  seven  individual  alternatives,  combines 
three  of  the  German  alternative  locations  into  a  single  entry,  and  does  not  include  the 
baseline  (“status  quo”)  alternative. 

14.  Tie  benefits/effectiveness  to 

mission  need 

Score  3  -  Partially  Met 

In  DOD’s  AOA  body  of  work,  the  AOA  team  provides  a  general  explanation  of  how 
mission  needs  will  or  will  not  be  met  under  each  of  DOD’s  AOA  criteria.  However,  the 

AOA  body  of  work  does  not  provide  evidence  that  the  AOA  team  developed  specific 
measures  of  effectiveness  or  tied  those  measures  to  specific  mission  needs. 

15.  Develop  life-cycle  cost  estimates 
Score  2  -  Minimally  Met 

According  to  our  best  practice,  the  life-cycle  cost  estimate  should  include  all  costs  from 
inception  of  the  effort — in  this  case,  JIAC  consolidation — through  design,  development, 
construction,  operation,  maintenance,  and  disposal.  However,  the  documentation  in 

DOD’s  AOA  body  of  work  does  not  provide  the  information  required  for  full  life  cycle  cost 
estimates.  Instead,  the  AOA  team  developed  partial  life  cycle  cost  estimates  as  part  of  the 
AOA  process. 

16.  Include  a  confidence  interval  or 
range  for  LCCEs 

Score  1  -  Not  Met 

The  AOA  body  of  work  provides  evidence  that  the  AOA  team  estimated  alternatives’  costs 
and  savings  as  point  estimates.  However,  the  body  of  work  does  not  provide  evidence 
that  the  AOA  team  calculated  a  life  cycle  cost  analysis  for  each  alternative  with  a 
confidence  interval  or  range.  Further,  the  body  of  work  does  not  provide  evidence  that  the 
AOA  team  used  a  method  of  comparing  alternatives  that  presents  a  comparable  view  of 
the  risk  associated  with  each  alternative. 

17.  Perform  sensitivity  analysis 

Score  3  -  Partially  Met 

Given  that  four  of  the  five  AOA  criteria  used  by  DOD  in  its  AOA  process  were  non- 
quantitative,  it  may  have  been  difficult  for  the  AOA  team  to  apply  sensitivity  analysis  to 
these  four  criteria.  However,  in  regard  to  DOD’s  fifth  AOA  criterion  on  cost,  the  AOA  body 
of  work  does  not  provide  evidence  that  the  AOA  team  tested  or  documented  the 
sensitivity  of  the  cost  and  benefit  or  effectiveness  estimates  for  each  alternative  to  risks 
and  changes  in  key  assumptions.  Further,  DOD  officials  stated  that  the  AOA  team  did  not 
conduct  such  an  analysis  for  its  cost  estimates. 

IV.  Document  and  review  the  AOA  process 

18.  Document  AOA  process  in  a  single 
document 

Score  2  -  Minimally  Met 

DOD’s  AOA  body  of  work  contains  one  summary  document:  the  201 1  preliminary  AOA 
summary.  However,  the  201 1  preliminary  AOA  summary  contains  neither  all  of  the 
information  in  DOD’s  AOA  body  of  work  nor  all  of  the  information  required  by  AOA  best 
practices. 

19.  Document  assumptions  and 
constraints 

Score  3  -  Partially  Met 

The  body  of  work  provides  evidence  that  the  AOA  team  considered  and  documented 
assumptions  and  constraints.  However,  the  body  of  work  does  not  provide  evidence  that 
assumptions  and  constraints  are  detailed  and  justified  for  each  alternative  in  the  AOA 
plan.  Instead,  they  are  generally  discussed  by  analysis  criterion. 

20.  Ensure  AOA  process  is  impartial 
Score  4  -  Substantially  Met 

Key  documents  in  the  AOA  body  work  provide  evidence  that  the  AOA  team  conducted  its 
analysis  without  a  predetermined  solution.  While  the  AOA  team  developed  the 
recommendation  to  consolidate  the  JIAC  at  RAF  Croughton,  this  recommendation  was 
not — during  the  AOA  process — DOD’s  final  decision. 
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Best  Practices  and  Four  Characteristics,  With 

GAO’s  Evaluation  of  the  Department  of 

Defense’s  (DOD)  AOA  Process 

Best  practices  for  the  AOA  process 

Summary  of  GAO’s  analysis 

21.  Perform  independent  review 

Score  3  -  Partially  Met 

The  body  of  work  provides  some  evidence  that  DOD’s  AOA  process  partially  met  certain 
requirements  of  this  best  practice.  For  example,  there  is  evidence  that  an  independent 
review — such  as  a  review  of  the  AOA  process  to  identify  viable  alternatives — occurred  at 
an  early  stage  of  the  process.  Also,  after  the  AOA  process  moved  from  EUCOM  into  DOD 
higher  headquarters  organizations  and  review  processes  (such  as  the  Basing  Directorate 
in  the  Office  of  the  Secretary  of  Defense  and  DOD’s  programming  and  budgeting  project 
approval  process),  DOD  entities  independent  of  the  core  AOA  team  reviewed  analysis 
and  recommendations  produced  by  the  core  AOA  team.  However,  the  body  of  work  does 
not  provide  evidence  that  DOD’s  AOA  process  involved  an  independent  review  to  ensure 
all  best  practices  were  followed  or  that  the  AOA  process  was  completed  with  enough 
thoroughness  to  ensure  that  an  independent  organization  outside  of  the  project’s  chain  of 
command  could  review  the  AOA  documentation  and  clearly  understand  the  process  and 
rationale  that  led  to  the  selection  of  the  recommended  alternative. 

V.  Select  a  preferred  alternative 

22.  Compare  Alternatives 

Score  2  -  Minimally  Met 

DOD’s  AOA  body  of  work  does  not  provide  evidence  that  the  AOA  team  used  net  present 
value — as  required  by  this  best  practice — to  compare  alternatives.  Instead,  DOD’s  body  of 
work  provides  evidence  that  the  AOA  team  used  two  other  methods  of  comparison — but 
did  not  compare  all  alternatives  to  each  other — and  does  not  explain  why  the  team  did  not 
use  net  present  value. 

Source:  GAO  analysis  of  DOD  information.  1  GAO-16-853 
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GAO  received  the 
Department  of  Defense’s 
comments  on  June  6, 
2016. 


UNCLASSIFIED 

OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 

3400  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3400 


ENERGY. 

INSTALLATIONS 
AND  ENVIRONMENT 

Mr.  Brian  J.  Lepore 
Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  D.C.  20548 


Dear  Mr.  Lepore, 


(U)  This  is  the  Department  of  Defense  response  to  the  GAO  draft  report,  GAO-16-563C, 
“JOINT  INTELLIGENCE  ANALYSIS  COMPLEX:  DOD  Did  Not  Fully  Use  Best  Practices  to 
Analyze  Alternatives  and  Should  for  Future  Military  Construction  Decisions,”  dated  April  27,  2016 
(GAO  Code  1 00388).  The  Department  previously  provided  technical  corrections  for  this  report  to 
your  staff  during  the  week  of  April  18,  2016.  While  we  appreciate  a  number  of  findings  enumerated 
in  the  draft  report,  the  Department  disputes  GAO’s  claim  that  its  “22  best  practices”  are  broadly 
applicable  to  a  wide  range  of  activities  that  includes  military  construction,  acquisition,  or  basing  and 
non-concurs  with  GAO’s  recommendation  that  the  Department  incorporate  those  practices  into  its 
military  construction  decision-making  process. 

(U)  The  Department  appreciates  GAO  documenting  the  failing  condition  of  facilities  at  RAF 
Molesworth,  UK  that  support  the  critical  intelligence  activities  of  the  U.S.  European  and  Africa 
Commands  and  our  allies.  The  current  operation  is  housed  in  21  disparate,  undersized,  World  War  II, 
Cold  War-era,  and  leased  temporary  facilities  that  are  in  poor  condition  and  are  at  the  end  of  their 
useful  life.  Recapitalization  is  absolutely  necessary  to  ensure  US  and  allied  forces  intelligence 
operations  are  supported  by  infrastructure  that  is  adequately  sized  and  configured  to  meet  mission 
requirements.  Doing  so  at  RAF  Croughton  through  construction  of  the  Joint  Intelligence  Analysis 
Complex  (JIAC)  makes  the  most  sense  both  operationally  and  financially. 

(U)  The  Department  also  appreciates  GAO’s  assessment  that  the  information  from  our 
Analysis  of  Alternatives  (AoA)  is  “sufficiently  reliable  for  the  purpose  of  describing  DoD’s  rationale 
for  choosing  RAF  Croughton  as  the  location  for  JIAC  consolidation.”  Of  the  14  locations 
considered,  RAF  Croughton  rated  highest  overall  against  our  five  evaluation  criteria:  impact  on 
intelligence  operations  (critical);  impact  on  bilateral  and  multinational  intelligence  collaboration 
(critical);  impact  on  international  agreements  and  relationships;  impact  on  community  quality  of  life; 
and  business  case.  Of  the  alternatives  that  met  the  critical  operational  requirements,  RAF  Croughton 
provided  the  best  business  case  as  it  is  an  enduring  base  whose  existing  housing,  support  facilities, 
available  land,  and  robust  communication  infrastructure  minimized  the  up-front  costs  of  this  project. 
Consolidating  at  RAF  Croughton  enables  return  to  the  UK  of  RAF  Molesworth  (and  the  associated 
support  site  at  RAF  Alconbury),  resulting  in  annual  savings  of  $74  million  that  will  allow  the 
Department  to  recover  its  investment  in  less  than  four  years  and,  thereafter,  enhance  readiness  by 
application  to  other  priorities. 
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(U)  Although  the  Department  also  appreciates  GAO’s  finding  that  in  arriving  at  the  decision 
to  recapitalize  this  critical  intelligence  facility  at  RAF  Croughton  the  Department  met  21  of  the 
GAO’s  “22  best  practices,”  the  Department  takes  issue  with  GAO  assessing  the  JIAC  basing  decision 
against  criteria  published  long  after  the  Department’s  analysis/decision  was  complete.  Furthermore, 
the  Department  does  not  accept  GAO’s  assertion  that  its  “22  best  practices”  are  universally 
applicable,  particularly  when  considering  basing  decisions  in  an  operational  theater  that  must  take 
into  account  a  number  of  subjective  factors  such  as  best  military  judgement. 

(U)  The  Department  therefore  non-concurs  with  GAO’s  recommendation  that  it  develop 
guidance  requiring  the  use  of  GAO’s  “22  best  practices”  for  military  construction  decisions.  As  is 
the  case  with  all  of  the  Department’s  military  construction  requests,  this  project  was  developed  and 
communicated  to  the  Congress  consistent  with  statute  and  the  Department’s  long-standing, 
supporting  policies.  Specifically,  Section  2802(c)  of  title  10,  United  Stated  Code,  requires  the 
Department’s  budget  submission  to  include  infonnation  on  “cost-effective  practices  as  an  element  in 
the  project  documents.”  In  support  of  that  requirement  the  DoD  Financial  Management  Regulation, 
Volume  2B,  Chapter  6,  requires  inclusion  of  a  DD  Form  1391  for  each  project  submitted  with  the 
budget  request,  and  that  each  DD  1391  contain:  results  of  an  economic  analysis  conducted  in 
accordance  with  DoD  Instruction  7041.3,  Economic  Analysis  for  Decision-making;  infonnation  on 
the  current  situation  (why  existing  facilities  are  unsuitable);  the  requirement  (why  the  project  is 
needed  and  what  alternatives  were  considered  -  including  use  of  existing  facilities);  and  the 
operational  impact  if  the  project  is  not  provided. 

(U)  The  Department  feels  strongly  that  Section  2802(c)  and  our  long-standing  supporting 
policies  provide  sufficient  information  to  support  military  construction  decisions,  associated  requests 
for  authorization/appropriation,  and  congressional  oversight.  In  cases  where  the  Congress  desires 
additional  infonnation  on  a  particular  project  beyond  that  which  existing  statute/policy  require,  it 
routinely  requests  and  receives  that  infonnation  through  reporting  or  certification  requirements  -  as 
was  the  case  with  this  project  in  the  National  Defense  Authorization  Acts  for  FYs  14-16.  Through 
this  process  the  Department  has  addressed  all  congressional  concerns  identified  to  date,  as  evidenced 
by  congressional  authorization  and  appropriation  of  two  (of  three)  project  phases  reviewed  thus  far. 


The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 


Sincerely, 


(Energy,  Installations  and  Environment) 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card,  MasterCard, 

Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 

Washington,  DC  20548 

Strategic  Planning  and 
External  Liaison 

James-Christian  Blockwood,  Managing  Director,  spel@gao.gov,  (202)  512-4707 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7814, 
Washington,  DC  20548 
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